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FOREWORD 


The  introduction  of  new  equipment  into  the  Army  inventory  poses  special 
challenges  for  reserve  component  (RC)  units.  RC  maintenance  units  frequently 
have  mobilization  missions  to  maintain  high-technology  weapon  systems  to  which 
they  have  little  access  for  training.  This  report  summarizes  a  computer-based 
training  (CBT)  program,  the  Model  Training  Program  for  Reserve  Component  units 
(MTP-RC),  designed  specifically  to  meet  RC  training  needs.  The  MTP-RC  program 
used  innovative  CBT  design  techniques  to  develop  courseware  for  training  RC 
soldiers  to  troubleshoot  and  maintain  Ml  tanks.  Also  included  is  a  summary  of 
the  MTP-RC  survey  on  industry-wide  CBT  development  costs  and  times. 


The  MTP-RC  was  a  joint  research  and  development  project  of  the  U.S.  Army 
Research  Institute  and  TRADOC’s  Training  Technology  Agency.  The  program  was 
directed  by  the  ARI  Fort  Knox  Field  Unit  as  part  of  a  memorandum  of  under¬ 
standing  dated  April  1983  with  Training  and  Doctrine  Command  (TRADOC),  U.S. 
Army  Forces  Command  (FORSCOM),  and  the  U.S.  Army  Research  Institute  (ARI). 
These  final  results  were  presented  to  the  project  sponsors,  TRADOC,  FORSCOM, 
and  the  Armor  and  Ordnance  schools  in  October  1987.  Plans  are  being  made  to 
convert  the  Army-wide  distribution  and  to  use  the  validated  MTP-RC  instruc¬ 
tional  approach  in  the  development  of  CBT  for  other  systems. 


EDGAR  M.  JOHNSON 
Technical  Director 


MODEL  TRAINING  PROGRAM  FOR  RESERVE  COMPONENT  UNITS:  FINAL  REPORT 


EXECUTIVE  SUMMARY 


Requirement : 

The  introduction  of  nev  equipment  into  the  Army  inventory  poses  special 
problems  for  training  in  the  Reserve  Component  (RC) .  Soldiers  in  RC  main¬ 
tenance  units  must  be  ready  to  support  the  equipment  in  times  of  mobilization, 
but  they  have  limited  time  for  training  and  often  have  little  or  no  hands-on 
experience  with  the  nev  equipment  during  their  weekend  drills  or  annual 
training. 

Computer-based  training  (CBT)  with  interactive  videodisc  technology  can 
deliver  training  with  realistic  video  images,  computer-generated  graphics,  and 
part-task  simulations.  This  medium  has  the  potential  to  train  and  sustain 
skills  on  new  equipment,  thereby  reducing  the  time  needed  for  training  after 
mobilization.  The  Model  Training  Program  for  Reserve  Components  Units  (MTP- 
RC)  was  conceived  by  the  U.S.  Army  Research  Institute  and  the  Training  and 
Doctrine  Command’s  Training  Technology  Agency  to  evaluate  the  effectiveness 
and  difficulties  of  developing  and  delivering  CBT  for  RC  units. 

Procedure: 

Computer-based  training  material  ("courseware")  was  developed  for  four 
Military  Occupational  Specialties  responsible  for  troubleshooting  and  main¬ 
tenance  on  the  Ml  Abrams  tank,  a  weapon  system  that  had  been  fielded  at  only 
one  RC  unit  at  the  time  of  the  program.  The  courseware  was  fielded  at  three 
RC  units  for  1-year  trials,  during  which  time  soldiers  took  the  Ml  training  on 
the  computer  system.  Hands-on  and  written  pre-  and  posttests  were  conducted 
to  evaluate  the  training  and  transfer  effects  of  the  training.  The  difficul¬ 
ties  of  determining  courseware  procurements,  estimating  development  costs, 
developing  large  amounts  of  courseware,  and  fielding  the  courseware  at  RC 
units  were  also  investigated.  A  survey  of  over  200  developers  of  CBT  was 
conducted  to  make  recommendations  about  analyzing  and  costing  courseware 
projects. 


Findings: 

The  training  and  transfer  evaluation  showed  that  computer-based  training 
can  be  effective  in  training  troubleshooting  and  maintenance  skills  in  RC 
units  with  little  access  to  real  equipment,  limited  expertise  on  the  equip¬ 
ment,  and  limited  time  for  training. 

The  experiences  of  the  MTP-RC  project  and  those  reported  in  the  survey  of 
CBT  developers  suggested  some  solutions  to  the  difficulties  encountered  in 
producing  and  fielding  the  courseware. 


•  Statements  of  work  for  courseware  development  projects  require  more 
detailed  descriptions  of  soldier  performance  objectives  and  desired 
training  features  than  do  statements  for  traditional  training. 

•  There  are  no  widely  accepted  measures  to  use  in  specifying  the  quan¬ 
tity  or  quality  of  courseware  to  be  developed  or  delivered.  Phased 
procurements,  in  which  developers  first  conduct  a  thorough  training 
analysis  and  then  attempt  to  define  the  scope  and  nature  of  the 
courseware,  should  be  investigated. 

•  For  developing  large  quantities  of  CBT  courseware,  enhancements  to  the 
Systems  Approach  to  Training  can  result  in  higher  quality  of  course¬ 
ware  and  increased  production  efficiency. 

RC  units  are  already  faced  with  a  heavy  schedule  of  skill  training  and 
sustainment.  For  CBT  to  be  successfully  implemented  in  RC  units,  the  program 
must  have  Command  support,  training  and  support  for  computer  operators,  and  an 
environment  suitable  for  housing  the  computer. 


Utilization  of  Findings: 

The  results  of  this  project  can  provide  guidance  for  analyzing,  costing, 
developing,  and  fielding  computer-based  training  in  the  Reserve  Component  and 
the  Regular  Army.  Consideration  of  the  recommendations  made  in  this  report 
can  help  enhance  some  of  the  potential  and  avoid  some  of  the  problems  inherent 
in  CBT. 


PROJECT  SCOPING  AND  PROCUREMENT  . 

Development  of  CBT  Requires  a  More  Detailed  Statement 

of  Work  Than  Traditional  Training  Does  . 

Scope  and  Design  of  the  Courses  in  a  Program  Can  Only 

Be  Clarified  After  Analysis  . 

Skills  and  Knowledge  Must  be  Specified  for  Objectives 

and  Entry  Level  Skills  . 

Phased  CBT  Procurement  Should  Be  Studied  . 

Statements  of  Work  for  CBT  Should  Include  Detailed 

Descriptions  of  Courseware  . 

Student  Objectives  Should  Be  Emphasized  in  the  Statement 

of  Work  . 

COURSEWARE  COSTING  . 

A  Three-Part  Study  of  Courseware  Costing  Was  Conducted  . 

Developers  Have  No  Widely  Endorsed  Measure  for  Courseware  . 

Developers’  Cost  Estimates  Are  Usually  Inaccurate  . 

Developers  Use  Several  Methods  and  Factors  When  Estimating  Costs  .  .  . 

Common  Factors  Affected  MTP-RC  Development  Costs  . 

Further  Research  on  Costing  Is  Recommended  . 

Measures  of  Courseware  Should  Emphasize  Student  Achievement  . 

COURSEWARE  PRODUCTION  . 

Common  Courseware  Production  Methods  Focus  on  Individual  Lessons  .  .  . 

Steps  of  MTP-RC  Production  Are  Described  . 

The  Analysis  and  Design  Phases  Were  Enhanced  with  MacroDesign 

on  This  Program  . 

Use  of  Lesson  Templates  in  the  Design  and  Development  Phases 

Improved  Quality  and  Increased  Efficiency  . 

Reviewing  MacroDesign  and  Templates  Can  Be  Difficult  . 

Recommendations  Are  Given  for  Use  of  Templates  for  Courseware 

Production  . 
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MODEL  TRAINING  PROGRAM  FOR  RESERVE  COMPONENT  UNITS: 

FINAL  REPORT 


INTRODUCTION 


The  Model  Training  Program  for  Reserve  Component  units  (MTP-RC)  program 
was  a  research  and  development  program  sponsored  by  the  Army  Research  Institute  (ARI) 
and  Training  and  Doctrine  Command’s  (TRADOC)  Training  Technology  Agency  to 
evaluate  computer-based  training  (CBT)  as  a  means  of  delivering  training  to  Reserve 
Component  (RC)  soldiers.  The  purposes  of  this  report  are  to  describe  MTP-RC, 
summarize  previous  reports  on  the  program,  and  describe  the  lessons  learned  about  the 
development  and  application  of  computer-based  training  for  training  in  the  RC 
environment. 

The  target  audience  for  the  training  developed  under  this  program  was  Reserve 
Component  soldiers  with  responsibility  for  troubleshooting  and  maintenance  of  the  Ml 
Abrams  tank.  The  program  is  a  response  to  the  problem  that  RC  soldiers  have  limited 
opportunity  for  training  and  sustaining  skills  on  new  equipment  such  as  the  Ml.  The 
training  was  developed  for  soldiers  in  four  specific  Military  Occupational  Specialities 
(MOSs): 

45E  Ml  Turret  Mechanic 

45K  Turret  Repairman 

63E  Ml  Automotive  Mechanic 

63H  Automotive  Repairman. 

The  syllabus  developed  for  this  program  included  five  courses.  The  first  course  was  a 
general,  introductory  course  on  the  Ml  for  all  four  MOSs.  It  included  tank  familiarization, 
safety,  technical  manuals  (TMs),  and  Ml  Test,  Measurement,  and  Diagnostic  Equipment 
(TMDE).  The  other  four  courses  were  specific  to  critical  tasks  of  each  of  the  four  MOSs. 
The  tasks  emphasized  were  electronic  troubleshooting  and  mechanical  maintenance  of  the 
systems  on  the  tank. 

The  potential  value  of  using  CBT  to  deliver  training  to  the  Reserve  Component  was 
emphasized  by  the  fact  that,  although  all  of  these  soldiers  would  have  responsibdities  for 
support  of  Ml  battalions  upon  mobilization,  most  of  them  had  little  or  no  training  or 
hands-on  experience  on  the  Ml  tank.  In  addition,  they  have  little  opportunity  to  work  on 
the  tanks  during  their  monthly  or  annual  training  periods.  This  is  a  common  problem  as 
new,  sophisticated  weapon  systems  are  introduced  into  the  Army  inventory.  The  program 
developed  and  evaluated  realistic  part-task  simulations  supported  by  interactive  videodisc 
(IVD)  for  enhanced  retention  of  skills. 

The  courseware  was  fielded  at  three  RC  units  for  at  least  one  year  at  each  site. 
Effectiveness  of  the  training  was  evaluated  by  pre-  and  posttests  of  soldier  performance  on 
written  and  hands-on  tests. 

The  overall  technical  goal  of  the  program  was  evaluation  of  computer-based 
training  with  IVD  as  a  training  medium  for  the  Reserve  Component.  Recommendations 
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were  to  be  made  about  the  implementation  of  CBT  for  RC  training.  To  this  end,  the  main 
technical  objectives  of  the  program  were  to: 

develop  computer-based  courseware 
investigate  courseware  production  efficiencies 
investigate  costs  of  courseware  production  and  delivery 
revise  the  courseware  based  on  formative  evaluation,  and 
validate  the  training  effects  of  the  courseware. 

Of  special  interest  throughout  this  report  are  the  results  of  a  major  study  done  in 
this  program  to  investigate  methods  and  successes  of  estimating  costs  for  development  of 
computer-based  training.  An  exhaustive  review  of  available  resources  on  CBT  costing  was 
conducted  as  well  as  a  survey  of  over  200  organizations  which  develop  CBT.  This  studv  of 
cost  estimation  provided  insight  into  the  problems  that  face  both  purchaser  and  developers 
of  CBT  today.  Information  from  this  study  is  integrated  with  the  discussions  of  the  MTP- 
RC  program  in  this  report  to  provide  a  broader  perspective  on  CBT  implementation. 

This  Report  Describes  the  MTP-RC  Program  and  Makes  Recommendations. 

Several  reports  have  already  been  submitted  and  published  on  research  and 
development  done  on  this  program.  This  final  report  will  summarize  the  results  of  these 
previous  reports  and  present  additional  salient  points  not  within  the  scope  of  those  reports. 
Many  of  the  problems  encountered  and  lessons  learned  on  this  program  will  be  applicable 
to  other  CBT  military  training  projects  beside  Reserve  Component  training. 

This  report  presents  the  experiences  of  MTP-RC  and  makes  recommendations  in 
five  main  sections: 

Scoping  and  Procurement, 

Courseware  Costing, 

Courseware  Production, 

Fielding  and  Implementation,  and 
Evaluation  of  MTP-RC  Courseware. 


PROJECT  SCOPING  AND  PROCUREMENT 

The  survey  of  CBT  developers  was  conducted  to  investigate  on  a  wider  scale  the 
scoping  difficulties  suggested  by  the  MTP-RC  experience.  As  a  research  and  development 
project  that  required  development  of  a  substantial  amount  of  CBT  courseware,  the  MTP- 
RC  program  offered  unique  opportunities  to  investigate  the  process  of  scoping  and 
procuring  CBT  development.  In  the  original  statement  of  work,  the  descriptions  of  the 
courseware  to  be  developed  had  to  be  stated  broadly  because  the  SOW  included  both 
research  into  the  RC  training  needs  and  investigation  of  advanced  CBT  capabilities  as 
tasks  prior  to  courseware  design.  This  flexibility  in  courseware  specifications  usually  is  not 
intended  in  procurements  which  are  exclusively  for  CBT  development,  but  changes  in 
scope  and/or  content  have  been  found  to  be  the  norm  nonetheless.  As  the  MTP-RC 
courseware  specifications  evolved  based  on  the  initial  research  &  investigation  of  the 
program,  valuable  lessons  were  learned  in  how  procurements  of  computer  based  training 
would  benefit  from  statements  of  work  with  more  detail  and  prior  training  analysis.  The 
survey  showed  that  problems  with  scoping  the  development  effort  are  endemic  to 
interactive  courseware  projects  today.  Common  difficulties  include  defining  the  size  of  the 
project,  describing  the  desired  features  of  the  courseware,  selecting  the  skills  and 
knowledge  to  be  included  in  the  training,  specifying  the  use  of  available  media,  and 


adjusting  the  breadth  and  depth  of  coverage.  Clear  definitions  of  these  aspects  of  the 
procurement  are  difficult,  but  at  some  point  in  the  planning  and  conducting  of  a 
development  effort  these  parameters  must  be  set.  Until  the  course  scope  is  based  on 
accurate  and  detailed  knowledge  of  the  skills  and  knowledge  to  be  trained,  the  training 
audience,  institutional  constraints,  and  resource  limitations,  there  will  continue  to  be 
changes  in  the  syllabus,  the  design,  the  budget,  the  schedule,  or  all.  These  changes 
frequently  result  in  delays  and  increased  costs.  A  cooperative  effort  between  the 
Government  procuring  agency  and  the  contractor,  with  clearly  defined  responsibilities  in 
the  effort,  is  required  to  obtain  all  the  information  needed  to  accurately  scope  CBT 
projects.  This  section  will  examine  these  problems  and  make  recommendations  on  how  the 
Government  and  contractors  can  cooperate  in  the  procurement  process  and  development 
stages  to  meet  the  Government’s  training  needs. 

Although  adjustments  and  classifications  of  the  courseware  development  were 
expected  in  this  research  effort,  the  MTP-RC  scoping  process  will  be  examined  to  show  the 
difficulties  which  are  also  encountered  on  programs  which  are  not  expected  to  change. 

This  section  will  describe  the  MTP-RC  scoping  starting  with  release  of  the  SOW,  how  the 
scope  changed  as  more  thorough  analysis  was  conducted,  problems  caused  by  the  changes, 
and  recommendations  for  improvements  in  the  procurement  process  that  could  benefit" 
both  the  Government  and  contractors.  The  basic  problem  is  that  until  a  thorough  training 
analysis  is  conducted,  it  is  not  possible  to  establish  what  the  course  will  be  like,  what  it  will 
cost,  how  long  it  will  take  to  develop,  or  whether  the  budget  is  adequate  for  the  training 
needs. 


The  statement  of  work  (SOW)  for  the  Model  Training  Program  required 
development  of  computer-based  training  for  four  MOSs  with  Ml  maintenance 
responsibilities.  Organizational  MOSs  45  E  and  63E  are  specific  for  troubleshooting  and 
maintenance  of  the  Ml  turret  and  hull,  respectively.  Direct  support/general  support  MOSs 
45K  and  63H  are,  respectively,  turret  and  hull  repairmen  with  responsibility  for  the  Ml 
among  other  vehicles.  MOS  45K  and  63H  soldiers  with  school  training  on  the  Ml  have  the 
Additional  Skill  Indicators  (ASI)  L-8.  Training  was  to  be  developed  to  sustain  skills  in 
tasks  already  learned  at  the  schools  and  to  build  skills  in  tasks  that  are  learned  after  gaining 
more  experience. 

The  content  of  the  courseware  would  thus  focus  on  these  Ml-related 
responsibilities.  After  award  of  the  contract,  the  specific  tasks  to  be  trained  were  selected 
from  the  MOS  critical  task  lists  by  ARI  with  assistance  from  the  Ordnance  School  and 
Armor  School.  The  tasks  were  mainly  troubleshooting  and  repairing  specific  electronic 
subsystems,  the  turbine  engine,  and  the  transmission  of  the  tank.  The  contractor  then 
conducted  a  training  task  analysis  on  these  tasks  to  determine  the  enabling  skills  and 
knowledge  for  these  tasks  and  whether  the  RC  soldiers  possessed  these  enabling  skills. 

This  process  of  selecting  tasks  for  the  course  content  based  on  their  research  value 
meant  that  the  syllabus  was  still  in  flux  several  months  into  the  project.  This  was 
acceptable  and  expected  on  this  research  contract.  However,  the  survey  of  developers 
showed  that  this  process  is  common  on  regular  development  projects.  This  means  that  the 
precise  content  of  the  course  usually  was  not  known  to  either  its  sponsors  or  the  contractor 
at  the  time  it  was  funded. 

The  amount  of  courseware  to  be  developed  was  expressed  in  the  commonly  used 
measure  of  instructional  hours.  The  SOW  called  for  development  of  approximately  240 
hours,  60  hours  for  each  MOS.  The  lessons  were  to  be  organized  in  four-hour  modules  to 


fit  the  typical  training  time  available  during  weekend  drills.  The  difficulty  with  hours  of 
courseware  was  emphasized  in  the  study  of  cost  estimating.  Although  commonly  used, 
most  developers  dislike  the  measure  because  it  gives  little  clear  definition  as  to  the  size  of 
the  effort.  An  "hour  of  instruction"  might  be  the  time  required  for  an  average  student  to 
complete  some  training,  might  include  remediation,  might  include  optional  helps,  and 
might  include  feedback  messages.  The  nature  of  an  "hour"  is  very  imprecise  and  SOW’s 
which  use  it  suffer  from  its  lack  of  precision. 

The  features  desired  in  the  courseware  were  described  in  the  SOW  as  interactive 
simulations,  videodisc,  gaming,  and  computer-generated  graphics.  The  applications  of 
these  features  and  amounts  ot  each  were  to  be  determined  in  the  design  phase,  based  on 
the  analysis  of  the  content  and  audience.  Like  most  statements  of  work,  the  MTP-RC 
SOW  did  not  specify  complexity  of  branching,  amount  or  complexity  of  graphics,  simulation 
fidelity,  types  of  remediation  or  help,  or  depth  of  treatment  ot  the  content.  The  SOW  did 
include  a  task  for  the  contractor  to  recommend  and  design  a  courseware  management 
system  to  keep  records  of  student  progress. 

Scope  and  Design  of  the  Courses  in  a  Program  can  only  be  Clarified  After  Analysis. 

Another  result  of  the  changing  content  of  the  courseware  procurements  was  that 
some  of  the  content  required  design  and  presentation  strategies  different  from  those 
expected  on  the  basis  of  the  Request  for  Proposals  (RFP).  For  example  the  MTP-RC  RFP 
required  development  of  a  sample  storyboard  for  a  representative  task  which  turned  out  to 
be  very  different  from  the  actual  tasks  selected  to  be  taught  in  the  CBT  courses  after  the 
RC-M1  skills  were  investigated.  The  sample  task  was  a  simple  mechanical  repair 
procedure,  but  the  criteria  used  for  selecting  tasks  once  the  project  started  explicitly 
eliminated  such  simple  tasks  an  1  focused  on  complex  electronic  troubleshooting 
procedures  and  advanced  mechanical  procedures  unique  to  the  Ml  equipment. 

The  survey  of  developers  indicated  that  it  is  a  common  problem  in  CBT 
procurements  to  have  changes  in  content  which  require  changes  in  design.  The  content 
and  its  required  design  strategies  are  frequently  unknown  at  the  time  of  bidding,  or  change 
once  the  project  starts.  The  result  is  usually  either  changes  in  the  syllabus,  changes  in  the 
budget,  or  both.  In  many  cases  discrepancies  between  the  expected  and  training  tasks 
meant  that  a  contractor  had  bid  development  of  one  type  of  content,  but  had  to  develop 
content  which  required  very  different  features,  and  therefore,  had  a  very  different  cost. 

The  design  developed  for  the  courseware  after  analysis  of  the  tasks  and  the 
audience  included  several  features  deemed  necessary  to  help  develop  the  target  skills. 

Some  of  the  features  were: 

o  long,  detailed  part-task  simulations  of  procedures, 
o  extensive  graphics  to  represent  tank  components  in  the  simulations, 
o  motivational  graphics,  such  as  cartoons, 
o  extensive  on-line  help  for  procedures, 
o  detailed  descriptions  of  system  operations, 
o  use  of  graphics,  video  or  both  on  most  screens  (Very  few  screens 
were  all  text.),  and 

o  on-screen  symbols  or  "icons"  for  soldiers  to  mark  with  the  light 
pen  to  move  through  the  courseware. 

When  the  contractor  conducted  the  training  analysis  for  courseware  design  and 
development,  it  became  clear  that  it  would  not  be  possible  for  soldiers  to  receive  thorough 
training  of  all  of  the  selected  tasks  in  the  limits  of  the  total  courseware  hours,  nor  would  it 
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be  possible  for  the  contractor  to  develop  that  much  courseware  with  the  time  and  budget 
allotted.  As  a  result,  the  syllabus  was  adapted  and  the  number  of  tasks  for  each  MOS  was 
reduced. 


Another  problem  with  the  critical  task  list  was  discovered  during  the  analysis  and 
design  phases  conducted  by  the  contractor.  The  entire  list  could  not  be  included  in  the 
course  because  of  time  and  budget  limitations.  The  training  analysis  resulted  in  a  decision 
to  include  knowledge  about  operation  and  function  of  the  subsystems  the  soldiers  would 
troubleshoot.  The  complex  systems  required  substantial  lessons  to  explain  their  functions. 
In  addition,  the  selected  Ml  troubleshooting  tasks  were  often  several  hours  long,  and  their 
part-task  simulations  were  almost  as  long.  As  a  result,  the  number  of  systems  to  be  covered 
had  to  be  reduced. 

These  changes  of  content  and  design  of  the  target  courseware  on  MTP-RC,  where 
they  were  expected  are  typical  of  changes  on  other  projects  where  they  were  not  expected. 
Surveyed  developers  indicated  that  changes  in  the  scope  or  design  features  occur  on  many 
projects  and  are  one  of  the  most  commonly  cited  causes  of  cost  overruns. 


Skills  and  Knowledge  Must  be  Specified  for  Objectives  and  Entry  Level  Skills. 


During  the  contractor’s  survey  of  Reserve  Units,  it  became  apparent  that  most  of 
the  Reserve  Component  soldiers  had  not  completed  school  courses  in  these  MOSs  or  ASIs. 
(The  ASI  for  the  Ml  for  63H  and  45K  is  L-8.)  Therefore,  they  did  not  possess  the  Mi- 
related  skill  levels  that  had  been  assumed  by  the  statement  of  work.  This  meant  that  the 
training  had  to  include  basic  skills  such  as  using  the  Ml  Test,  Diagnostic,  and  Measurement 
Equipment  (TMDE),  locating  components  on  the  tank,  and  using  the  special  Ml  technical 
manuals.  (The  technical  manuals  used  the  Skill  Performance  Aid.)  This  need  was  the 
impetus  for  the  development  of  a  common  course  for  all  MOSs  to  familiarize  them  with 
the  tank  and  its  equipment.  This  was  a  major  component  of  the  syllabus  that  had  not  been 
anticipated  in  the  statement  of  work  or  the  proposal.  This  addition  to  the  course  content  of 
the  program  required  that  some  of  the  originally  desired  content  had  to  be  deleted  from 
the  syllabus  in  order  to  stay  within  the  project  budget.  This  experience  of  MTP-RC  is 
common  in  CBT  procurements  because  the  desired  course  is  specified  before  a  thorough 
training  analysis  is  complete. 

Phased  CBT  Procurement  Should  be  Studied. 


The  lesson  learned  is  the  MTP-RC  course  specifications  evolved  and  the 
information  gained  from  the  survey  of  CBT  developers  leads  to  several  recommendations 
for  the  CBT  procurement  process.  The  possibility  of  phased  procurements,  in  which  the 
analysis  phase  is  completed  before  the  following  phases  are  awarded,  should  be  examined. 
This  would  permit  a  clearer  understanding  and  firmer  specifications  for  the  effort. 

Methods  for  bidding  and  procuring  CBT  contracts  should  be  studied  to  see  how  the 
methods  might  be  modified  to  facilitate  more  accurate  specifications  of  the  needed 
courseware.  One  suggestion  made  frequently  in  the  survey  and  supported  by  the  MTP-RC 
experience  is  to  conduct  the  analysis  phase  of  the  Systems  Approach  to  Training  (SAT) 
before  developing  the  requirements  for  the  design  and  development  phases.  A  thorough 
analysis  of  the  content,  the  audience,  and  the  resources  available  would  provide 
information  necessary  to  generate  the  scope,  design,  and  cost  of  courseware  development 
efforts. 
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In  some  cases  the  analysis  may  be  completed  by  the  purchasing  agency,  although  the 
developer  will  generally  need  at  least  a  complete  review  of  the  analysis,  and  possibly  a 
partial  analysis  to  gather  information  not  provided  by  the  purchaser.  In  other  cases,  the 
contractor  may  perform  the  analysis  as  a  separate  contract  prior  to  the  procurement  of  the 
development  project. 

Without  a  completed  training  task  analysis,  the  choice  of  media  for  the  project  is 
made  without  sufficient  information.  A  thorough  analysis  can  support  more  iniormed 
media  selection  in  two  ways.  Given  the  training  tasks  identified  by  the  analysis,  the  most 
effective  medium  or  media  for  the  type  or  types  of  tasks  can  be  selected.  For  example, 
well-founded  recommendations  on  the  advisability  of  video  or  computer-generated 
graphics  can  be  made  only  when  these  tasks  are  analyzed.  On  the  other  hand,  if  the  main 
selection  of  media  has  already  been  made  by  the  using  agency,  the  analysis  will  allow  the 
selection  of  tasks  for  training  that  can  be  most  successfully  trained  with  that  medium. 

Another  advantage  of  conducting  the  analysis  separately  is  that  it  would  allow  the 
Government  to  make  choices  about  the  scope  of  the  project  and  the  design  of  the 
courseware  based  on  more  complete  information.  The  initial  analysis  should  provide 
detailed  data  on  the  real  skill  deficiencies  and  provide  options  for  training  designs  with 
reliable  cost  estimates  for  finished  courseware.  Without  this  information,  both  the 
Government  and  the  contractor  are  left  to  make  estimates  without  sufficient  foundation. 
For  example,  after  analysis  of  the  troubleshooting  tasks  and  the  knowledge  and  skill  levels 
of  the  RC  soldiers,  the  contractor  proposed  a  training  design  that  included  knowledge  of 
the  functions  of  the  system  components,  an  introduction  to  the  troubleshooting  logic 
behind  the  procedure  for  the  symptom,  a  guided  demonstration  on  a  part-task  simulation, 
and  practical  exercises  on  a  part-task  simulation.  Only  then  did  it  become  clear  that  the 
budget  of  the  contract  was  not  sufficient  to  allow  such  thorough  treatment  of  all  the 
selected  tank  troubleshooting  tasks.  Had  the  analysis  work  been  done  previously  as  a 
discrete  phase,  the  Government  would  have  had  the  option  to  choose  a  less  expensive 
design,  train  fewer  systems,  or  expand  the  program. 

In  any  interactive  courseware  development,  there  will  be  decisions  that  involve 
trade-offs,  such  as  effectiveness  of  media  versus  availability  of  media,  depth  of  content 
coverage  versus  breadth  of  coverage,  use  of  more  realistic  video  images  versus  less  realistic 
computer  graphics,  and  so  forth.  These  decisions  are  often  changed  or  renegotiated  during 
the  contract  as  the  training  analysis  uncovers  details  of  the  needed  training.  Survey 
respondents  listed  changes  in  the  scope  of  work  as  one  of  the  most  common  causes  of 
inaccurate  estimates  of  development  costs  and  lost  time  during  production. 

Caution  must  be  used  in  separating  the  analysis  from  the  development  project. 
Separate  contracts  with  the  Government  could  lead  to  a  lengthy  procurement  process. 
Additionally,  in  cases  where  the  purchasing  agency  provides  the  analysis  to  the  developer, 
care  must  be  taken  to  ensure  that  all  of  the  information  that  a  CBT  developer  might  need 
is  provided  or  can  be  gathered.  Toward  this  end,  a  checklist  of  information  to  be  provided 
in  Requests  for  Proposals  (RFPs)  for  development  could  ensure  that  essential  standard 
information  be  supplied  or  that  a  task  of  the  contract  allows  for  this  information  to  be 
collected  before  tne  development  effort  is  scoped.  A  standard  analysis  should  include 
objectives  and  performance  criteria,  audience  entry  level  skills,  and  required  instructional 
features  or  strategies. 

Statements  of  Work  for  CBT  Should  Include  Detailed  Descriptions  of  Courseware. 

Even  without  the  thorough  training  analysis  suggested  above,  the  results  of  the 
survey  indicate  that  more  detailed  descriptions  of  the  courseware  desired  would  allow  both 
the  Government  and  the  contractor  to  overcome  some  of  the  difficulties  of  unspecified 
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products  and  time-consuming,  expensive  revisions.  For  purchasers  who  must  make  changes 
to  existing  training  or  develop  training  from  scratch,  specifying  the  desired  number  of 
lessons  or  student  objectives  with  descriptions  of  the  level  of  complexity  of  branching  and 
graphics  may  be  the  most  helpful.  For  example,  a  developer  or  purchaser  might  specify: 

Four  simulations  with  linear  branching;  two  types  of  help;  with  50%  video  images, 
40%  computer-generated  graphics,  and  10%  all  text  pages;  with  meaningful 
interaction  on  50%  of  the  pages;  to  meet  the  objective  of  enabling  the  soldier  to 
identify  the  causes  of  four  common  faults  of  the  stated  system. 

If  the  procuring  agency  does  not  have  the  experience  with  interactive  courseware  to  make 
these  specifications,  they  should  get  assistance  from  other  agencies  or  contract  the  task  of 
writing  specifications.  The  specifications  will  be  done,  either  explicitly  or  implicitly,  for 
every  CBT  project.  Explicit  specifications  allow  more  intelligent  choices  about  use  of 
resources,  and  the  sooner  they  are  done,  the  sooner  the  choices  can  be  made. 

Completion  of  the  analysis  and  even  a  prototype  lesson  before  the  entire  project  is 
bid  will  ensure  that  most  issues  are  clearer  to  both  parties.  Prototype  lessons  could  be  used 
to  display  exactly  what  features  are  required  and  what  type  of  courseware  will  satisfy  the 
training  requirement. 

tudent  Objectives  Should  be  Emphasized  in  the  Statement  of  Work. 


Soldier  performance  of  troubleshooting  and  maintenance  tasks  was  always 
emphasized  as  the  goal  of  the  MTP-RC  courseware,  but  the  survey  of  developers  found 
that  on  other  projects  the  emphasis  is  sometimes  equivocal.  Developers  reported  that 
purchasers  are  sometimes  so  fascinated  with  the  techniques  and  features  of  the  CBT 
medium  that  the  goal  of  final  student  achievement  is  overshadowed. 

In  the  flurry  of  discussion  about  defining  CBT  development  efforts,  the  most 
important  question  is  often  ignored:  Do  the  soldiers  learn?  There  are  no  generally 
accepted  standards  for  courseware  qualify,  but  certainly  a  major  component  of  any 
definition  is  the  students’  achievement  of  the  objectives.  The  measures  of  courseware  and 
tools  to  estimate  development  costs  must  be  built  around  the  issue  of  quality  to  make 
realistic  estimates  and  fairly  compare  bids.  Two  developers  can  bid  for  the  same  number 
of  instructional  hours  with  very  different  processes.  The  question  that  must  be  addressed 
is:  Will  the  students  learn  equally  well?  To  this  end,  courseware  developers  may  find  it  an 
advantage  to  guarantee  results  rather  than  a  specific  number  of  "instructional  hours." 

Many  of  the  comments  from  developers  in  the  survey  indicated  that  failure  to  specify 
student  outcomes  clearly  contributes  to  ineffective  courseware  or  excessive  revision.  The 
focus  of  a  request  for  proposals  should  be  on  providing  all  information  necessary  to  ensure 
that  the  description  or  the  desired  student  outcomes  are  detailed  and  unambiguous. 


desired  student  outcomes  are  detailed  and  unambiguous. 


The  focus  on  student  achievement  has  other  advantages  as  well.  Developers  and 
purchasers  would  learn  to  distinguish  between  strategies  and  features  that  help  the  student 
and  those  which  are  merely  fashionable.  A  focus  on  the  objectives  could  also  reduce  the 
number  of  revisions,  as  only  changes  that  assist  the  student  in  meeting  the  objectives  need 
be  made. 

This  focus  on  student  achievement  rests  on  a  very  critical  task-the  ability  of 
developers  and  purchasers  to  determine,  at  the  beginning  of  the  project,  exactly  what 
outcome  for  students  is  desired.  After  communicating  with  over  300  CBT  professionals, 
the  MTP-RC  researchers  believe  that  clearly  specifying  student  outcomes  would  make  the 
single  most  important  contribution  to  the  ability  of  developers  to  estimate  development 
efforts  accurately  and  produce  computer-based  training  that  teaches. 
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COURSEWARE  COSTING 


A  Three-Part  Study  of  Courseware  Costing  was  Conducted. 

A  major  research  focus  of  the  MTP-RC  program  was  to  address  concerns  expressed 
by  purchasers  and  developers  of  computer-based  training  that  current  cost  estimating 
methods  for  courseware  development  are  glaringly  inadequate.  Accurate  cost  estimates  are 
equally  essential  for  both  purchasers  and  developers  of  CBT.  Purchasers  must  have 
reliable  methods  of  estimating  CBT  costs  so  they  have  a  realistic  basis  for  deciding  whether 
CBT  is  cost-effective  for  particular  training  applications.  Key  factors  which  contribute  to 
the  cost  of  CBT  must  be  clearly  defined  and  weighted  for  purchasers  to  evaluate  proposals. 
CBT  developers  have  also  expressed  an  acute  need  for  effective  cost-estimating  tools,  since 
methods  for  accurately  estimating  CBT  development  costs  are  an  essential  prerequisite  for 
successful  project  planning  and  management.  A  three-part  study  was  undertaken  to 
investigate  the  current  state  of  CBT  development  costing  methods  and  make 
recommendations  for  improvements  in  the  current  process.  The  study  is  reported  by  Jay, 
Bernstein,  and  Gunderson  (1987). 

This  study  consisted  of  three  parts,  each  designed  to  contribute  to  the  attainment  of 
the  above  goals.  Part  One  consisted  of  a  review  of  the  issues  involved  in  estimating  CBT 
development  costs.  Sources  of  information  for  this  section  included  available  literature, 
research  in  progress,  CBT  professionals,  and  anecdotal  information  from  seven  completed 
CBT  projects.  Part  Two  consisted  of  a  survey  of  almost  200  CBT  developers  to  determine 
cost  estimating  practices,  identify  and  rank  factors  impacting  development  costs,  and 
collect  data  on  average  CBT  development  times.  Part  Three  included  a  review  of  cost 
models  for  CBT  and  the  related  field  of  software  development,  as  well  as  an  informal 
validation  of  an  existing  CBT  development  costing  tool.  The  major  findings  of  the  study 
are  summarized  here.  Additional  details  on  the  methods  of  the  survey  and  the  resulting 
data  are  included  in  the  interim  research  report. 

Developers  Have  No  Widely  Endorsed  Measure  for  Courseware. 

Developers  reported  that  no  standard  method  for  measuring  CBT  is  currently  used. 
Although  the  "instructional  hour"  is  most  commonly  used,  it  is  widely  disliked  by 
developers  as  inaccurate  and  not  reflecting  complexity  of  content,  complexity  of  branching, 
quality,  and  other  factors.  Purchasers  using  this  method  have  little  assurance  that  an  "hour 
of  instruction"  will  teach  the  objective  or  even  that  it  will  actually  take  an  hour.  Although 
the  purchaser  or  developer  may  have  in  mind  a  number  of  hours  they  would  like  the 
instruction  to  take,  it  is  often  impossible  to  predict  delivery  time  in  the  computer  based 
medium. 

In  addition,  developers  reported  including  different  tasks  when  costing  a  unit  of 
CBT,  so  when  purchasers  are  reviewing  proposals,  they  cannot  be  sure  that  the  proposals 
will  produce  the  same  courseware.  The  exclusion  of  certain  tasks,  such  as  analysis, 
formative  evaluation,  and  management,  is  likely  to  have  a  major  effect  on  the  quality  and 
effectiveness  of  the  courseware.  Since  the  effectiveness  of  courseware  is  seldom 
guaranteed,  and  the  development  tasks  are  inconsistent,  the  review  process  may  be  the 
client’s  only  way  to  impact  quality.  Many  professionals  who  contributed  to  this  research 
would  welcome  both  a  standard  method  for  measuring  CBT  that  includes  quality  factors 
and  a  method  for  making  estimates  of  CBT  development  times.  Most  believed  that 
developing  measurement  and  estimation  tools  would  be  complex,  but  valuable. 


Most  developers  report  that  they  are  unable  to  make  accurate  estimates  for  CBT 
development  times.  Less  than  10%  are  able  to  estimate  within  5%  of  actual  cost. 
Experienced  developers  are  no  better  overall  at  making  accurate  estimates,  but 
inexperienced  developers  are  more  often  off  by  over  20%.  However,  there  is  evidence  that 
experienced  estimators  of  development  times  are  better  at  making  estimates  than  those 
only  experienced  in  CBT  production.  Because  of  poor  estimates,  many  developers  are  only 
breaking  even  or  actually  losing  money  on  CBT  development  contracts.  De-scoping  and 
requests  for  additional  funds  are  the  norm,  and  purchasers  often  do  not  receive  what  they 
expected.  However,  there  is  an  indication  that  developers  can  improve  estimates  by 
considering  project  complexity  factors  when  making  estimates. 

There  are  many  reasons  for  inaccurate  bids.  There  is  no  standardized  estimation 
method  and  few  historical  records  are  readily  available.  Even  when  historical  data  do  exist, 
developers  may  be  reluctant  to  use  it  because  the  numbers  are  unacceptably  high  in  light 
of  "industry  averages,"  and  they  believe  that  future  projects  would  not  have  similar 
problems.  In  addition,  developers  are  often  surprised  by  factors  not  anticipated  in  the 
beginning,  such  as  changes  in  scope  by  the  client  or  hardware  or  software  changes. 

Developers  generally  bid  on  CBT  development  projects  with  very  little  knowledge 
of  the  project  details.  Developers  often  underbid  projects  because  they  fail  to  anticipate 
factors  such  as  the  required  complexity,  quantity  of  courseware,  or  long  review  processes. 
Some  developers  are  now  insisting  on  a  complete  analysis  and  initial  design  before  the 
development  project  is  bid  to  insure  that  most  of  the  underlying  issues  are  identified 
initially  and  taken  into  account  when  making  a  bid. 

The  survey  of  nearly  200  organizations  which  develop  computer-based  training 
found  that  accurate  estimation  of  CBT  development  times  is  not  yet  common  in  the 
industry.  Among  the  major  finding  are  the  following  points. 

o  Only  9%  of  private  organizations  reported  estimating  costs  within  5%. 

o  Experienced  developers  do  not  report  any  lower  development  times  than 
do  inexperienced  developers. 

o  Inexperienced  developers  are  twice  as  likely  as  experienced  developers  to  be  off 
in  estimates  by  20%  or  more. 
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o  For  respondents  using  the  instructional  hour  as  the  unit  of  measure,  the  mean 
minimum  development  time  for  one  hour  of  courseware  was  140  hours; 
the  mean  maximum  was  3 16  hours. 

o  The  range  of  development  times  for  the  instructional  hour  was  one 
to  4,000  hours. 

o  Different  developers  include  different  development  tasks  in  their  estimates  for  a 
unit  of  courseware. 

o  The  seven  highest  rated  cost  factors  were,  in  order:  complexity  of  instructional 
design  strategy,  nature  and  complexity  of  content,  client’s  demand  for 
revisions,  complexity  of  screen  interactions,  complexity  and  volume  of  graphics, 
conditional  branching  complexity,  and  development  team’s  experience 
producing  CBT. 
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o  The  most  frequently  used  single  measure  of  courseware  is  the  instructional  hour, 
but  only  1%  continue  to  recommend  it  use. 


Developers  reported  use  of  several  methods  of  estimating  the  cost  of  a  unit  of 
instruction.  Most  commonly,  "industry  averages”  of  100  to  400  hours  per  hour  of  instruction 
are  used.  The  reported  range  of  hours  required  to  develop  a  unit  of  CBT  was  from  1  to 
4000  hours  per  instructional  hour,  with  153  and  316  as  the  mean  low  and  high  in  the  survey. 
Although  this  is  close  to  the  industry  average  of  100  to  400  hours  per  hour,  these  broad 
ranges  are  virtually  useless  for  estimation  without  a  systematic  way  of  narrowing  the  range 
for  a  specific  project.  Developers  simply  must  be  able  to  estimate  within  10%  of  cost  if 
profit  is  an  issue.  A  few  developers  use  their  own  historical  databases  to  predict  costs  for 
projects,  and  a  few  others  are  developing  costing  tools  that  include  cost  factors. 

A  host  of  factors  were  identified  during  the  research  as  contributing  to  CBT 
development  costs.  The  factors  mentioned  most  often  and  rated  as  having  the  greatest  cost 
effect  are: 

o  Complexity  of  instructional  design  strategy, 
o  Clarity  of  project  specifications  at  outset  and  adherence  to  them, 
o  Complexity  of  content, 

o  Number  of  revisions/unexpected  client  revisions, 
o  Complexity  and  number  of  features  (e.g.,  graphics  and  helps)  and, 
o  Experience  of  the  development  team,  individually  and  as 
as  a  group. 

Although  several  attempts  have  been  made  to  quantify  these  factors  to  yield  a  unit 
estimate,  none  has  been  proven  any  more  accurate  than  applying  the  industry  averages. 

One  commercially  available  tool  for  estimating  CBT  development  costs,  The  CBT 
Analyst  (Park  Row  Software,  1986),  was  found  to  be  useful  for  novices  who  need  an 
estimate  of  the  cost  range  to  make  a  decision  about  whether  to  go  ahead  with  CBT. 
However,  like  other  guidelines  currently  available,  the  tool  cannot  be  used  with  confidence 
to  predict  the  cost  of  any  particular  project.  The  tool  does  point  developers  in  the  right 
direction  by  forcing  them  to  focus  on  the  factors  that  can  increase  costs. 

The  more  mature  field  of  software  development  offers  promise  for  the  future 
development  of  an  accurate  CBT  development  costing  tool.  Because  of  similarities 
between  software  and  courseware  development,  it  seems  reasonable  to  apply  to  courseware 
the  techniques  used  for  measuring  and  developing  software.  However,  research  is 
necessary  before  tool  development  can  begin.  Recommendations  for  the  steps  in  creating  a 
CBT  development  estimation  tool  are  presented  later  in  this  section. 

Common  Factors  Affected  MTP-RC  Development  Costs. 

Of  the  seven  factors  identified  in  the  survey  of  developers  as  most  affecting 
development  time,  several  had  large  impact  on  MTP-RC  production,  and  are  described  as 
follows.  The  factors  which  most  affected  the  cost  of  courseware  production  for  the  MTP- 
RC  were  discussed  in  the  report  "An  Analysis  of  Cost  of  Computer-Based  Training 
Hardware  and  Courseware  Development  for  the  Model  Training  Program  for  Reserve 
Component  Units"  (Begg  and  Bernstein,  1986). 


because 


o  The  instructional  design  strategies  developed  in  the  design  phase  were  complex 
e  part  of  the  purpose  of  the  program  was  to  test  the  capabilities  of  CBT  to  deliver 
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highly  interactive  training  and  part-task  simulations.  There  were  several  types  of 
interactions,  a  number  or  presentation  strategies  used  for  each  lesson,  and  extensive 
feedback.  This  increased  the  time  required  for  production. 

o  The  content  of  the  course  was  highly  technical  information  about  the  electronic 
and  mechanical  systems  of  the  Ml  tank.  Extensive  research  and  writing  time  was  required 
of  the  Subject  Matter  Experts  (SMEs),  as  were  considerable  interactions  with  the 
Instructional  Designers  (IDs),  Graphics  Specialists  (GSs),  and  Video  Specialists  (VSs)to 
assure  accuracy  of  the  course  content. 

o  The  screen  interactions  were  fairly  complex  in  that  several  branching  options 
were  available  from  every  screen.  Even  with  the  lesson  code  templates,  many  hours  of 
coding  and  writing  by  the  Courseware  Developers  (CDs)  were  necessary. 

o  The  strategy  for  training  and  testing  troubleshooting  skills  was  especially  heavy  in 
its  use  of  graphics.  The  use  of  motivational  cartoons  and  detailed  functional  diagrams, 
which  added  to  the  acceptance  and  effectiveness  of  the  courseware,  also  added  to  the  hours 
required  for  developing  graphics. 


o  Although  several  of  the  contractor’s  personnel  had  previously  worked  on  CBT 
projects,  the  MTP-RC  project  was  their  first  effort  as  a  team.  It  took  time  for  the  team 
members  to  develop  communication  paths  and  working  patterns.  The  previous  experience 
as  a  team  was  also  cited  in  the  survey  of  developers  as  a  major  factor  in  production  time. 


Although  detailed  production  figures  were  not  kept  for  all  lessons  throughout  the 
production  phase,  some  fairly  accurate  estimates  can  be  given  for  the  hours  required  to 
produce  some  lessons.  Table  1  shows  the  average  numbers  of  hours  required  in  four  labor 
categories  for  each  of  the  five  lesson  types  when  the  process  of  production  with  templates 
was  in  place  and  running  smoothly.  Figures  for  Video  Specialists  and  Production 
Coordinators  are  not  included  on  the  table  because  they  could  not  be  apportioned  to 
individual  lessons.  These  figures  pertain  only  to  lessons  developed  with  the  templates  after 
the  template  design  and  coding  were  stable  because  the  earlier  lessons  included  time  for 
developing  and  revising  the  generic  templates  in  addition  to  coding  the  specific  lessons. 


Table  1 

MTP-RC  Courseware  Production:  Average  Labor  Figures  in  Hours 


Lesson  type  (number  of  lessons! 


Labor 

category 


Name, 
Location, 
Function  (6) 


Input, 
Process 
Output  (4) 


Intro  (5) 


Trouble¬ 

shooting 

(6) 


Mainten- 
*ance  (4) 


ID 

110 

110 

40 

120 

155 

CD 

100 

100 

60 

155 

125 

GS 

50 

50 

40 

80 

50 

SME 

30 

30 

20 

55 

75 

Note.  All  figures  are  +  /-  20%. 
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The  difference  in  costs  for  the  five  types  of  lessons  was  mainly  due  to  the  complexity  of  the 
design  and  complexity  of  the  technical  content  of  the  troubleshooting  and  maintenance 
lessons. 


Further  Research  on  Costing  is  Recommended. 

Many  of  the  approximately  300  CBT  professionals  who  participated  in  this  study  of 
CBT  cost  estimation  indicated  that  they  would  welcome  a  tool  to  standardize  the 
measurement  of  courseware  and  estimation  of  development  costs.  However,  before  such  a 
tool  can  be  developed,  a  number  of  key  elements  must  be  resolved,  including  data 
collection  procedures,  methods  of  CBT  measurement,  and  identification  of  key 
development  factors.  The  recommendations  are  listed  below. 

Further  research  is  needed  to  identify  and  validate: 

o  the  most  effective  means  of  collecting  and  reporting  data, 
o  the  most  effective  means  of  measuring  CBT, 
o  standard  descriptors  for  CBT, 

o  factors  and  weights  that  affect  CBT  development  costs,  and 
o  a  method  for  measuring  CBT  quality. 


Development  data  from  CBT  projects  should  be  collected  in  an  accessible  database 
to  be  used  on  bidding  future  projects  of  a  similar  nature.  Project  managers  should  note 
major  factors  affecting  cost  increases  or  decreases  for  future  reference.  A  government 
requirement  of  a  more  detailed  accounting  of  development  labor  would  provide  externa) 
motivation.  This  accounting  might  require  that  labor  hours  be  reported  for  each  lesson  and 
task  as  opposed  to  labor  category,  as  is  usually  required. 

Research  should  be  conducted  to  structure  and  validate  a  cost  estimating  tool.  To 
develop  an  accurate  cost  estimating  tool,  factors  believed  to  impact  the  cost  of  CBT 
development  effort,  including  quality,  must  be  identified.  Scales  should  be  developed  for 
each  factor  and  be  tested  for  reliability.  An  example  of  an  objective  scale  for  the  factor  of 
programmer  experience  would  be  0-2  years,  3-5  years,  and  6  or  more  years.  Sample 
completed  projects  would  then  be  selected  and  rated  on  each  factor’s  scale. 
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A  multilinear  regression  could  be  used  to  identify  the  factors  that  are  significantly 
correlated  with  project  development  time.  The  factors  would  then  be  validated  by  testing 
them  with  the  original  sample,  and  finally  with  a  new  sample.  Once  a  cost  model  has  been 
generated,  the  rules  and  weights  associated  with  could  be  embedded  into  a  computer 
program. 

Validated  tools  for  software  estimation  have  evolved  over  several  years.  The 
development  of  a  comprehensive  and  standardized  tool  for  estimating  CBT  development 
costs  is  also  like  to  take  several  years  to  complete.  In  the  meantime  developers  require 
methods  for  measuring  CBT  and  estimating  development  times.  For  purchasers  who  must 
make  changes  to  existing  training  or  who  are  developing  training  from  scratch,  lesson  or 
objectives  with  specifications  for  level  of  complexity  of  branching  and  graphics  may  yield 
the  most  helpful  measurements.  Using  this  method,  for  example,  apurchaser/developer 
might  specify:  "Four  simulations  with  linear  branching;  two  tyqjes  of  help;  with  50%  video, 
40%  graphic  and  10%  all  text  pages;  with  a  meaningful  interaction  on  50%  of  the  pages;  to 
meet  the  objectives  of  enabling  tne  student  to  identify  the  causes  of  four  common  faults  of 
the  stated  system  with  90%  of  the  students  achieving  90%  accuracy." 
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Purchasers  who  are  converting  existing  classroom  training  to  CBT  without 
significant  modifications  may  find  that  the  instructional  hour  can  still  serve  as  a  helpful 
measurement.  In  this  case  the  CBT  hour  would  be  equivalent  to  the  instruction  given  in 
one  hour’s  time  in  the  classroom.  However,  developers  and  purchasers  should  be  aware 
that  those  using  the  instructional  hour  have  found  it  to  be  the  source  of  many 
misunderstandings. 

Until  an  accurate  costing  model  has  been  validated,  developers  need  a  more  narrow 
range  for  estimating  an  hour  than  the  industry  average.  Those  who  do  not  have  a  historical 
data  base  may  find  it  useful  to  use  a  method  suggested  by  Gery  (1986)  for  examining 
courseware  factors  and  applying  development  ranges.  Caution  should  be  used  at  the  higher 
ranges  of  development  hours.  While  Gery  lists  300+  per  hour  as  the  highest  range, 
developers  have  reported  development  hours  of  400  to  4000  hours  per  hour  when  certain 
factors  are  present. 

New  developers  should  remember  that  there  are  some  tasks  which  must  often  be 
done  on  a  CBT  project  that  are  not  part  of  the  price  of  all  of  the  units  of  courseware.  These 
tasks  and  other  costs  must  be  added  to  the  estimate  for  the  development  of  CBT  to  ensure 
an  accurate  estimate. 

Measures  of  Courseware  Should  Emphasize  Student  Achievement. 

In  conducting  the  research  on  courseware  costing,  it  became  apparent  that  the  most 
important  question  is  often  ignored:  Do  the  students  learn?  There  are  no  generally 
accepted  standards  for  courseware  quality,  but  certainly  a  major  component  of  any 
definition  is  the  students’  achievement  of  the  objectives.  The  measures  of  courseware  and 
tools  to  estimate  development  costs  must  be  built  around  the  issue  of  quality  to  make 
realistic  estimates  and  fairly  compare  bids.  Two  companies  can  bid  for  the  same  number  of 
instructional  hours  with  very  different  prices.  The  question  that  must  be  addressed  is:  Will 
the  students  learn  equally  well?  To  this  end,  courseware  developers  may  find  it  an 
advantage  to  guarantee  results  rather  than  a  specific  number  of  "instructional  hours." 

Many  of  the  comments  from  developers  indicated  that  failure  to  clearly  specify 
student  outcomes  contributes  to  ineffective  courseware  or  excessive  revisions.  The  focus  of 
the  RFP  should  be  on  providing  all  information  necessary  to  ensure  that  the  description  of 
the  desired  student  outcomes  are  detailed  and  unambiguous. 

The  focus  on  student  achievement  has  other  advantages  as  well.  Developers  and 
purchasers  would  learn  to  distinguish  between  strategies  and  features  that  help  the  student 
and  those  which  are  merely  fashionable.  A  focus  on  the  objectives  could  also  reduce  the 
number  of  revisions,  as  only  changes  that  assist  the  student  in  meeting  the  objective  need 
be  made.  In  the  MTP-RC  review  process,  some  requests  for  changes  were  simply 
suggestions  for  alternative  strategies  or  presentations  that  did  not  directly  address  the 
effectiveness  of  the  lesson. 

This  focus  on  student  achievement  rests  on  one  very  critical  task--the  ability  of 
developers  and  purchasers  to  determine  at  the  beginning  of  the  project,  exactly  what 
outcomes  for  the  student  are  desired.  The  MTP-RC  statement  of  work  was  like  most 
SOWs  described  by  developers  in  that  it  did  not  specify  soldier  performance  outcomes  of 
the  training  program.  This  ambiguity  led  to  some  delays  and  extra  costs.  After 
communicating  with  almost  300  CBT  professionals,  the  researchers  believe  that  clearly 
specifying  student  outcomes  would  make  the  single  most  important  contribution  to  the 
ability  of  developers  to  estimate  costs  accurately  and  produce  computer-based  training  that 
teaches. 
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Computer-based  training  development  is  a  highly  competitive  field,  with  many 
factors  complicating  the  development  process.  Currently  many  developers  are  finding  it 
difficult  to  make  a  profit,  partly  due  to  their  inability  to  make  accurate  estimates,  and 
partly  due  to  a  misunderstanding  by  all  involved  of  the  complexity  and  opportunities  for 
cost  far  above  industry  averages.  Experience  in  the  field,  as  in  the  field  or  software 
development,  will  undoubtedly  improve  the  ability  of  developers  and  purchasers  to  work 
together  to  develop  effective  training.  The  recommendations  in  this  section  represent 
essential  steps  for  defining  and  systematizing  the  complex  process  of  estimating  CBT 
development  costs.  When  they  are  implemented,  these  recommendations  could  bring  about 
improvements  in  the  cost  estimating  process  which  will  be  a  boon  to  CBT  developers  and 
purchasers  alike. 


COURSEWARE  PRODUCTION 


As  the  contractor  planned  the  courseware  production  effort  for  MTP-RC,  it  became 
apparent  that  development  of  the  required  number  of  courseware  hours  within  the 
program’s  time  and  budget  constraints  would  be  impossible.  Production  of  CBT  typically 
follows  the  traditional  Instructional  Systems  Development  (ISD)  model,  setting  very 
general  design  standards  for  the  course  and  then  proceeding  lesson  by  lesson  through 
design,  development,  review,  and  revision.  This  approach  by  itself  was  not  considered 
likely  to  succeed  in  the  MTP-RC  constraints.  A  major  enhancement  of  the  process  was 
developed  for  analyzing  the  tasks  to  be  trained  because  the  contract  statement  of  work 
called  for  development  of  over  200  hours  of  courseware  in  a  relatively  short  time. 

Consequently,  the  contractor  applied  an  approach  to  analysis,  design,  and 
development  that  took  a  course-wide  approach,  making  global  decisions  that  were 
applicable  to  several  lessons  at  a  time,  instead  of  proceeding  immediately  to  development 
of  individual  lessons.  This  approach  was  found  to  be  effective  in  maintaining  quality 
standards  across  all  lessons  and  increasing  efficiency  of  development  of  complex  lessons. 

In  this  section  the  following  main  topics  will  be  discussed: 


o  shortcomings  of  common  CBT  production  methods  for  large 
development  projects, 

o  steps  of  the  MTP-RC  production  process, 

o  enhancements  to  the  analysis  and  design  phases  developed 
for  the  MTP-RC  production  effort, 

o  the  use  and  effects  of  lesson  templates  as  tools  for 
design  and  development  of  lessons, 

o  difficulties  with  reviewing  specifications  and  lessons  in 
the  templated  process,  and 

o  recommendations  for  effective  application  of  the  template 
approach. 
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Common  Courseware  Production  Methods  Focus  on  Individual  Lessons. 

In  developing  the  courseware  for  this  program,  the  Army’s  Systems  Approach  to 
Training  was  used,  and  the  standard  five  phases  were  implemented:  analysis,  design, 
development,  implementation,  and  evaluation.  The  SAT  has  been  very  effective  in  setting 
standards  for  development  of  training  materials  for  the  Army  with  its  comprehensive 
approach  for  curriculum  design  for  traditional  media,  such  as  lecture,  workbook,  and  shop 
floor  training.  However,  the  SAT  lacks  detailed  guidance  at  the  lesson  level  for  CBT  and 
provides  very  little  guidance  regarding  the  more  creative  uses  of  the  computer  as  a  training 
medium.  Thus,  computer  based  lessons  which  use  capabilities  such  as  simulations  and 
gaming  are  sometimes  designed  and  developed  without  a  systematic  approach  for  the 
entire  program. 

The  MTP-RC  design  team  therefore  set  about  to  enhance  the  SAT  approach  with 
three  main  goals  in  mind.  The  first  was  to  develop  guidelines  for  analyzing  the  training 
tasks  so  that  the  capabilities  of  CBT  could  be  more  effectively  applied  to  the  training 
objectives.  This  led  to  development  of  categories  of  objectives  and  instructional  designs  for 
each  category.  The  second  goal  was  to  provide  systematic  decision-making  guidelines  for 
selection  of  content,  instructional  strategies,  and  media  for  individual  lessons.  The  third 
goal  was  to  provide  design  tools  and  techniques  specifically  for  use  with  non-linear 
instructional  media. 

It  was  hoped  that  these  guidelines  and  tools  would  facilitate  communication  of 
lesson  content  and  programming  logic.  The  resulting  enhanced  design  process  was  called 
"MacroDesign"  because  it  involved  the  application  of  design  techniques  and  tools  at  the 
course  or  "macro"  level  as  well  as  at  the  lesson  level. 

The  implementation  of  this  MacroDesign  approach  and  an  evaluation  of  its 
effectiveness  are  presented  after  the  steps  of  production  are  described. 

Steps  of  MTP-RC  Production  are  Described. 

The  process  used  to  develop  MTP-RC  courseware  conforms  to  the  Instructional 
Systems  Development  model.  In  the  early  phases,  content  selection,  task  analysis,  and 
training  task  analysis  were  conducted  by  the  design  team.  The  design  process  also  added 
the  development  of  lesson  templates  as  described  above. 

Production  Labor  Categories.  Five  main  labor  categories  were  involved  in  the 
production  of  lessons.  Instructional  Designers  developed  the  overall  plan  for  the  design  of 
the  learning  activities.  They  selected  and  analyzed  content  for  inclusion  in  the  courseware 
and  then  designed  the  instruction  and  the  interactions  between  the  soldier  and  the 
courseware.  In  performing  these  tasks,  the  IDs  consulted  with  Subject  Matter  Experts 
about  content,  with  Courseware  Developers  about  programming  code,  and  with  the 
Government  about  their  requirements.  When  the  lesson  templates  were  completed,  the 
IDs  wrote  content  for  the  individual  lessons. 

Subject  Matter  Experts  provided  technical  expertise  on  the  tank  systems  and  proper 
performance  of  troubleshooting  and  maintenance  tasks.  In  addition,  the  SMEs  provided 
input  on  the  training  audience  and  reviewed  conformance  of  all  material  to  Army  doctrine. 

Courseware  Developers  did  the  programming  in  the  ADAPT  authoring  language  of 
the  MicroTICCIT  system  to  translate  the  IDs’  specifications  for  content  and  interactions 
into  working  computer  code.  The  senior  CDs  also  developed  the  code  for  the  prototype  for 
each  template. 
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Graphic  Specialists  created  the  technical  drawings,  charts,  menus,  diagrams,  and 
cartoons  used  to  illustrate  the  courseware  content.  For  the  MTP-RC  courseware,  many  of 
the  drawings  were  drawn  on  paper,  digitized  into  the  computer  memory,  and  cleaned  up 
using  the  on-line  graphics  editor.  A  more  advanced  graphics  package  was  added  to 
MicroTICCIT  near  the  end  of  this  program,  too  late  to  De  used  for  production. 

Video  Specialists  turned  the  designers’  requests  for  video  into  a  finished  videodisc. 
They  developed  the  storyboard  and  scripts  for  the  video  shoot,  supervised  the  location 
shooting,  and  directed  the  post-production  editing. 

Once  templates  were  designed,  coded,  and  approved,  the  development  phase  began 
and  production  of  lessons  commenced.  Most  of  the  lesson  development  was  performed  by 
the  prime  contractor.  Some  lessons  were  written  and  developed  by  a  subcontractor  and 
reviewed  by  the  prime  contractor  before  submittal  to  the  Government.  Some  other  lessons 
were  written  by  prime  contractor  designers  and  their  coding  and  graphics  were  produced  by 
the  subcontractor.  The  steps  in  the  development  phase  are  described  below. 

Step  1.  Write  lesson  specifications.  The  Instructional  Designers  wrote 
specifications  for  individual  lessons  for  approval  by  the  Government.  Since  the  lessons 
were  using  the  templates,  the  strategies  were  almost  the  same  for  each  lesson  of  a  type,  and 
the  differences  were  in  the  specific  content  and  the  emphasis  of  important  technical  and 
safety  points. 

Step  2.  Generate  lesson  content.  After  Government  approval  of  the  lesson 
specifications,  the  IDs  wrote  lessons  using  content  supplied  by  the  SMEs.  In  the  later 
stages  of  the  program,  SMEs  were  also  writing  lesson  content  to  conform  to  the 
instructional  strategies  of  the  appropriate  templates.  The  content  at  this  stage  included 
requests  for  graphic  and  video  images,  requests  for  unusual  branching,  special  windows  on 
the  images,  and  any  other  information  needed  to  produce  the  lesson. 

Step  3.  ID/SME  Review.  Lessons  were  then  submitted  to  another  ID  for  design 
review  and  to  a  SME  for  content  review.  Lessons  were  returned  to  the  author  for  revisions 
until  the  lesson  were  approved  for  production. 

Step  4.  Prepare  lesson  for  production.  When  the  lesson  content  was  approved,  the 
ID  submitted  it  to  the  Production  Manager,  who  channeled  the  various  components  of  the 
specifications  to  the  appropriate  production  personnel.  Graphics  requests  were  directed  to 
the  Graphic  Specialists  for  design  and  input  to  the  system.  Video  requests  were  sent  to  the 
Video  Specialist  who  turned  the  ID’s  requests  into  storyboards  and  scripts  for  the  video 
production.  The  CD  reviewed  the  lesson,  gathered  any  additional  information,  and 
prepared  the  lesson  for  entry  into  the  system.  The  technical  nature  of  most  of  the 
courseware  required  that  all  personnel  interact  frequently  with  the  SMEs,  who  served  as  a 
resource  throughout  the  program. 

Step  5.  Input  lesson  data.  In  this  phase,  the  specific  text,  graphics,  and  video  were 
entered  into  the  MicroTICCIT  system  for  the  lessons.  CDs  used  the  code  templates  as  the 
foundation  for  all  lessons,  but  it  was  usual  that  an  individual  lesson  would  require  some 
modification  to  the  template,  such  as  allowing  additional  pages  to  describe  a  system 
component  or  combining  graphics  and  video  images.  Modifications  were  made  as  the  data 
were  entered.  Graphics  Specialists  designed,  drew,  and  edited  the  various  graphics  and  put 
them  into  the  system. 

Step  6.  CD  review.  Each  lesson  was  reviewed  by  two  Courseware  Developers.  First, 
the  CD  who  created  the  lessons  verified  that  the  data  entry  was  accurate  and  that  any  code 
modifications  performed  as  intended.  Revisions  were  made  until  the  lesson  was  correct. 
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Then  a  second  CD  reviewed  the  lesson  independently  to  assure  that  the  lesson  worked 
properly:  branches  were  correct,  graphics  appeared  in  the  correct  place,  spelling  was 
correct,  etc.  Corrections  and  revisions  were  made  by  the  original  CD. 

Step  7.  ID  review.  When  the  CD’s  debugging  process  was  complete,  the  authoring 
ID  reviewed  the  lesson  on-line  to  assure  that  it  accurately  reflected  the  lesson 
specifications.  Text  or  graphics  sometimes  had  to  be  modified  by  the  CD  during  entry, 
usually  after  consultation  with  the  ID,  and  this  review  allowed  the  ID  to  confirm  that  the 
entire  lesson  was  effective.  This  was  not  a  major  design  review  for  two  reasons.  First,  the 
global  design  decisions  were  made  earlier  for  the  template,  and  second,  changes  in  the 
lesson-level  at  this  time  would  have  greatly  increased  development  time.  Revisions  were 
usually  made  by  the  original  CD. 

Step  8.  Senior  ID  Quality  Assurance  review.  The  senior  ID  reviewed  the  lesson  to 
assure  that  the  design  met  the  design  standards  and  lesson  specifications  approved  by  the 
Government. 


Step  9.  Video  production.  At  the  same  time  that  computer  code  and  graphics  were 
being  produced,  video  production  was  underway.  The  Video  Specialist  used  the  IDs’  lesson 
scripts  and  storyboards  to  develop  a  detailed  scripts,  shot  lists,  and  storyboards  for  the 
shooting  effort.  The  contractor  took  a  video  crew  to  the  Ordnance  School  and  Anniston 
Depot  to  tape  shots  of  tank  components,  procedures,  and  operations.  The  program 
schedule  and  the  video  budget  allowed  no  more  than  one  shooting  opportunity  at  each  site, 
and  this  was  scheduled  and  conducted  as  soon  as  possible.  As  a  result,  many  lessons  were 
developed  on-line  and  received  first  reviews  before  the  videodisc  was  ready.  As  with  most 
CBT  projects  that  involve  interactive  video,  to  have  waited  for  the  discs  to  produce  on-line 
and  review  was  not  possible  within  the  time  of  the  program. 

Step  10.  Army  review.  As  on-line  production  was  completed,  lessons  were  sent  for 
review  by  Government  representatives  at  the  Ordnance  School.  Since  the  major  designs 
for  templated  lessons  were  already  approved,  the  major  points  of  this  review  were  content 
accuracy  and  flow  of  the  lesson.  Army  SMEs  verified  correctness  of  all  technical  content 
and  procedures,  and  educational  specialists  reviewed  the  lesson  flow.  Requests  for 
corrections  and  revisions  were  sent  to  the  contractor.  The  schedule  of  video  production 
required  that  many  lessons  receive  preliminary  review  without  the  video  content. 

Step  11.  Integrate  video  and  overlay  graphics.  Video  shots  and  sequences  specified 
by  the  ID  were  integrated  into  the  courseware  by  the  CD.  Because  production  of  early 
lessons  had  to  be  concurrent  with  video  production,  those  early  lessons  had  the  video 
added  after  initial  review.  Other  lessons,  such  as  the  maintenance  template  lessons,  were 
so  dependent  on  the  video  that  they  could  not  be  written  until  the  videodiscs  were 
available.  Computer-generated  graphics  which  overlaid  video  displays  also  had  to  wait  for 
the  discs.  After  integration  of  the  video,  the  lessons  were  available  for  final  review  by  the 
Government. 


w<Zr.ryYjrJ 


]n  Phases  were  Enhanced  with  MacroDesian  on  this  Prc 


The  decision  was  made  to  analyze  the  training  tasks  for  similarities  across  tasks, 
hoping  to  find  commonalities  that  could  be  the  basis  for  standard,  repeatable  instructional 
designs.  As  depicted  in  Figure  1,  the  process  adds  to  the  standard  SAT  model  from  Step  3, 
where  enabling  objectives  are  clustered  by  similarities  for  development  of  generic 
objectives.  The  figure  shows  that  for  troubleshooting  and  maintaining  any  of  the 
subsystems  of  the  Ml  tank  (such  as  the  Computer  System  (CS),  Laser  Range  Finder  (LRF) 
or  Fire  Control  System  (FCS)),  the  enabling  objectives  include  describing  the  normal 
operations  of  the  system  and  checking  equipment.  For  these  common  enabling  skills  found 
in  the  analysis  of  troubleshooting  each  system,  generic  enabling  objectives  were  written, 
such  as  "describe  the  normal  operations  of  a  system." 

Since  these  generic  objectives  would  be  applied  many  times  throughout  the  course, 
one  effective  design  strategy  for  training  each  of  these  objectives  could  be  developed  and 
used  in  each  lesson  which  included  an  example  of  the  generic  objective.  The  following  five 
generic  objectives  were  identified  and  targeted  for  design  of  lesson  strategies: 

o  identify  the  name,  location,  and  function  of  the  components 
of  a  tank  system, 

o  identify  the  input,  process,  and  output  of  a  system,  component 

o  identify  abnormal  operations  of  a  system, 

o  perform  troubleshooting  inspections  and  tests  on  a  part-task  simulation,  and 

o  perform  mechanical  maintenance  tasks  on  a  part-task  simulation. 

Nearly  85%  of  the  content  of  the  entire  course  fit  comfortably  into  these  generic  objectives. 

With  these  objectives  identified,  the  senior  instructional  designers  on  the  program 
began  to  develop  instructional  strategies.  The  result  was  that  design  decisions  were  made 
consistently  across  the  course,  wherever  the  objectives  were  similar.  Of  particular 
importance  for  this  program  was  that  the  consistent  use  of  standard  designs  for  all  similar 
objectives  provided  the  Reserve  Component  soldiers  with  a  familiar  courseware  structure 
for  each  or  their  widely-separated  training  sessions.  If  the  courseware  had  required  the 
soldiers  to  become  familiar  with  different  types  of  objectives,  stated  differently,  for  each 
lesson,  much  of  their  monthly  4-hour  block  of  training  time  would  have  been  spent  in 
readjusting  to  the  computer  each  time.  The  standardization  of  the  lesson  designs  allowed 
implementation  of  a  standard  interface  for  the  courseware,  even  in  complex  simulations. 
The  standard  designs  for  the  lesson  types  were  called  templates,  which  will  be  described 
later  in  this  report. 

The  second  goal  of  enhancing  the  SAT  process  was  to  provide  guidelines  for  making 
design  decisions  at  the  lesson  level.  The  main  media  selection  had  been  made  for  this 
program:  the  courseware  would  use  a  computer-based  delivery  system  supported  by 
interactive  videodisc.  However,  that  global  decision  still  left  a  range  of  choices  for 
presentation  techniques  and  instructional  strategies  for  the  content  at  the  course  level  and 
the  lesson  level.  For  example,  should  a  particular  type  of  simulation  be  presented  as  video 
images  from  the  videodisc,  graphics  generated  by  the  computer,  or  a  combination  of  video 
image  with  computer-generated  overlay  graphics?  Should  an  operation  of  the  equipment 
be  shown  with  motion  video,  still  video,  animated  graphics,  or  still  graphics?  The  factors  to 
be  considered  included  instructional  effectiveness,  cost  of  the  production  choice, 
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preferences  of  the  using  schools,  and  a  host  of  other  considerations.  Although  these  types 
of  decisions  are  commonly  made  by  instructional  designers  in  developing  interactive 
courseware,  the  standard  SAT  process  gives  limited  guidance  on  selection  of  the  media 
globally,  and  virtually  none  at  the  lesson  level.  Faced  with  the  task  of  designing  and 
developing  over  one  hundred  lessons,  the  MTP-RC  development  team  realized  that  they 
did  not  have  the  time  to  make  these  decisions  from  scratch  for  each  lesson  or  module. 

Both  to  facilitate  the  decision-making  process  and  to  set  program-wide  standards  of 
design  quality,  the  designers  developed  a  set  of  guidelines  for  making  design  decisions  at 
both  the  course  and  lesson  level.  The  guidelines  were  presented  in  the  Appendix  to  the 
report  "An  Enhanced  Instructional  Design  Process  for  Developing  Interactive  Courseware." 
The  guidelines  which  were  found  to  be  especially  important  for  this  program  aided  in  the 
selection  of:  feedback  levels,  passing  criteria,  learner  control,  visual  fidelity,  simulation 
fidelity,  performance  error  fidelity,  and  level  of  guidance.  These  guidelines  set  standards 
for  selecting  options  for  each  of  these  issues  by  highlighting  the  impact  of  the  major  factors 
affecting  the  program  as  whole  and  the  lesson  under  development.  The  major  factors  for 
this  program  and  any  ISD  project  are:  the  audience  the  training  is  aimed  at,  the  institution 
sponsoring  the  training,  the  content  of  the  training,  and  the  resources  available  for  the 
development  and  delivery  of  the  courseware. 

These  guidelines  greatlv  simplified  and  facilitated  the  design  and  production 
process.  As  the  first  lessons  were  designed,  the  major  issues  were  identified  and  guidelines 
set  for  all  the  ensuing  lessons  of  that  type.  This  use  of  high-level,  or  global,  guidelines  for 
design  decisions  both  increased  the  quality  of  the  resulting  lessons  and  reduced  their  cost 
because  more  experienced  senior  designers  could  set  the  standards  that  would  be 
implemented  by  less  experienced  designers  on  all  lessons.  For  example,  it  was  decided  that 
for  mechanical  maintenance  procedures,  motion  video  sequences  would  be  used  only  when 
the  procedure  specified  a  particular  type  of  motion,  such  as  rotating  the  transmission. 
Simple  mechamcal  steps,  like  attaching  slings  to  the  the  transmission,  would  be  shown  with 
video  stills.  This  consideration  of  visual  fidelity  was  influenced  by  the  content  (the  task 
being  trained)  and  the  resources  available  for  production.  Motion  video  is  more  expensive 
to  produce,  and  the  video  budget  was  limited.  These  guidelines  aided  the  uniform 
implementation  of  -in  decisions  throughout  the  program.  Although  not  exhaustive  of 
all  consideration.^  se  guidelines  will  be  useful  to  designers  on  other  programs  because 

they  elucidate  mai  onsiderations  that  designers  have  often  had  to  make  either  intuitively 

or  fortuitously.  For  example,  the  guidelines  for  level  of  guidance  in  a  task  by  the  computer 
remind  the  designer  that  extensive  guidance  will  require  extensive  involvement  from 
subject  matter  experts;  if  this  resource  is  not  available,  extensive  guidance  may  not  be 
possible. 

Use  of  Lesson  Templates  in  the  Design  and  Development  Phases  Improved  Quality  and 


Increased  Efficiency. 

The  third  goal  in  enhancing  the  SAT  process  was  to  develop  tools  and  techniques 
for  both  content  writers  and  programmers  working  on  individual  lessons.  The 
MacroDesign  process  of  designing  courseware  as  described  above  resulted  in  detailed 
instructional  strategies  for  each  lesson  type.  For  each  of  these  strategies,  complete 
computer  code  was  planned  and  developed,  including  reusable  routines  or  "macros"  for 
presenting  screen  icons,  routines  for  branching,  routines  forjudging  and  recording  answers, 
and  all  of  the  other  code  required  for  a  complete  lesson.  These  programs,  along  with  the 
design  strategies  they  implemented,  together  make  up  lesson  "templates"-detailed, 
reusable  patterns  for  lessons  that  are  implementations  of  sophisticated  instructional 
strategies  for  types  of  lessons,  from  presenting  the  lesson  objectives  to  reporting  the  test 
score.  It  shoula  be  noted  that  although  it  is  now  common  for  courseware  programmers  to 
write  reusable  routines  for  interactions  of  portions  of  lessons,  it  is  uncommon  to  write 
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reusable  programs  for  entire  lessons.  The  ideal  for  this  endeavor  was  to  develop  program 
code  for  each  lesson  type  so  complete  and  flexible  that  the  programming  for  each 
individual  lesson  would  be  reduced  to  entering  content  and  data  specific  to  that  lesson. 

This  goal  could  not  be  reached,  but  approximately  85%  of  the  code  for  these  very  complex 
lessons  was  written  and  reusable  in  each  lesson.  Each  template  consists  of  a  fully 
prescribed  instructional  strategy,  a  set  of  screen  display  characteristics,  and  a  structured 
system  for  student  interaction. 

Instructional  strategies  were  developed  for  each  of  the  five  common  objective  types 
mentioned  above,  and  as  modules  were  developed  for  the  various  subsystems  of  the  tank, 
the  templates  were  implemented.  The  two  main  contributions  of  the  template  use  to  the 
MTP-RC  production  effort  were  that  the  templates  were  effective  in  ensuring  a  high  level 
of  integrity  in  the  instructional  design  and  program  coding  across  all  lessons,  and  they  did 
make  the  writing  and  coding  of  lessons  more  efficient  than  it  would  have  been  without 
them.  Several  of  the  templates  have  been  used  on  subsequent  CBT  programs  for  ARI  with 
only  minor  modifications.  However,  there  were  some  difficulties  in  using  them  which  will 
be  described  in  this  section. 

Reviewing  MacroDesian  and  Templates  can  be  Difficult. 

A  major  difficulty  encountered  in  using  MacroDesign  was  review  of  the  strategy  and 
designs  incorporated  in  the  templates.  It  is  a  typical  problem  in  CBT  design  that  the 
complex  interactivity  made  possible  by  use  of  the  computer  is  hard  to  portray  on  paper 
documents.  The  instructional  techniques  chosen  for  presenting  material,  providing 
practice,  and  conducting  tests  for  a  generic  objective  type  are  set  early  in  the  ISD  process. 
(In  fact,  the  MacroDesign  step  of  developing  templates  is  on  the  transition  between  the 
design  and  production  phases.)  These  design  templates  are  then  approved  by  the 
Government  before  lesson  production  begins.  The  difficulty  was  that  it  is  hard  to  envision 
the  application  of  a  single  instructional  strategy  pattern  for  a  number  of  lessons  which 
cover  apparently  disparate  content.  For  example,  the  same  strategy  (the  "Name,  Location, 
and  Function"  template)  was  used  to  teach  the  operations  of  the  computer  subsystem,  the 
transmission,  and  the  laser  range  finder  subsystem.  The  level  of  abstraction  required  to 
find  commonalities  across  these  objectives  is  somewhat  at  odds  with  visualizing  the 
concrete,  specific  interactions  provided  to  the  soldier  on  the  computer  screen.  One 
alternative  would  be  to  design  the  template,  develop  a  first  lesson,  and  use  that  lesson  as 
the  review  and  approval  vehicle.  However,  developing  a  first  lesson  requires  that  virtually 
all  of  the  template  coding  be  written  to  implement  that  lesson.  If  the  template  is  not 
approved,  then  substantial  time  and  money  will  have  been  lost.  A  smaller,  prototype  lesson 
can  be  developed,  but  if  the  prototype  is  small  enough  to  make  a  difference,  it  probably  will 
not  contain  enough  content  and  will  still  not  facilitate  generalizations  to  other  potential 
lessons  by  reviewers.  The  traditional  approach  of  storyboards  can  depict  the  presentation 
strategies,  but  breaks  down  when  depicting  the  interactions  possible  with  CBT  when  the 
branching  begins  to  get  even  a  little  complex. 


Review  of  the  first  MTP-RC  templates  unsuccessfully  used  a  software  engineering 
technique  called  "pseudocode."  Pseudocode  is  a  technique  for  describing  computer 
programs  that  uses  programming-like  descriptions  of  the  computer’s  activity,  structured  in  a 
flow  somewhat  like  the  actual  code  will  be  written,  but  using  natural  language.  This 
technique  worked  fairly  well  for  communicating  the  templates  to  programmers,  but  was  not 
successful  for  those  with  little  programming  background.  Most  reviewers,  designers  and 
especially  SMEs  have  no  programming  background  and  found  the  pseudocode  impressive 
but  not  helpful  for  understanding  the  proposed  interactive  instructional  strategies. 

The  later  templates  were  reviewed  through  a  combination  of  storyboards  and  small 
prototype  lessons  which  included  most  of  the  key  interactions.  This  technique  was  more 
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effective,  but  did  cause  some  delay  in  production.  Fortunately,  once  the  first  templates 
were  approved,  production  could  begin  on  lessons  of  that  type  while  templates  for  the 
other  types  were  being  developed. 

Some  problems  were  encountered  in  reviewing  lessons  developed  with  the 
templates.  Approval  of  a  template  requires  a  leap  of  faith  on  the  part  of  the  Government. 
The  templates  themselves  were  difficult  Lo  review  for  approval  of  the  instructional 
strategies  they  embodied  when  they  were  presented  on  paper.  Even  when  the  prototype 
lessons  were  presented,  some  reviewers  found  it  difficult  to  separate  the  generic  strategies 
from  the  lesson  as  implemented  with  specific  content.  Part  of  this  difficulty  arises  from  the 
fact  that  the  process  of  instructional  design  generally  involves  dealing  with  content 
abstractly,  for  example,  determining  whether  particular  content  is  a  skill  or  knowledge 
objective.  The  fact  of  life  is  that  most  personnel  with  the  technical  training  and 
background  to  evaluate  technical  accuracy  do  not  have  the  design  background  to  evaluate 
the  abstract,  formal  design,  and  vice  versa.  After  viewing  a  few  lessons,  most  reviewers 
were  able  to  provide  helpful  criticism  to  the  design  group,  but  after  several  lessons  are 
produced  with  templates,  the  major  changes  in  the  templates  would  defeat  the  purpose  of 
developing  production  efficiency,  because  they  would  have  to  be  rewritten. 

Based  on  the  experience  of  the  MTP-RC  program,  the  best  suggestion  for  reviewing 
the  templates  for  approval  of  design  would  involve  three  steps:  1)  provide  storyboards  for 
the  design  because  most  CBT  reviewers  will  be  familiar  with  that  format;  2)  produce  a 
prototype  lesson  with  sample  content  so  actual  interactions  can  be  seen  on  screen;  and  3) 
provide  extensive  briefings  to  all  reviewers  on  the  rationale  and  strategies  of  the  templates. 
The  need  for  communication  may  require  that  contractor  personnel  be  available  on-site  for 
extended  periods  to  discuss  decisions  made  for  individual  lessons  and  comments  made  on 
their  effectiveness. 

A  common  problem  with  military  CBT  development  is  that  SMEs  assigned  to  the 
project  will  often  rotate  to  new  duty  assignments  during  the  life  of  the  project  and  be 
replaced  by  new  SMEs  unfamiliar  with  the  overall  context  of  the  program  or  the  decisions 
made  early  in  the  program.  These  new  reviewers  must  have  thorough  briefings  on  the 
program  to  make  worthwhile  contributions  without  slowing  down  the  progress  of 
implementing  previous  decisions. 

Another  difficulty  faced  with  integrating  review  comments  into  lessons  is  that 
reviewers’  requests  for  apparently  simple  changes  in  the  screen  display  or  presentation  of 
information  may  in  fact  be  difficult  or  impossible.  This  problem  is  not  peculiar  to  the 
template  approach  of  production.  The  limitations  may  arise  from  previous,  global 
decisions,  from  lesson-specific  code  decisions,  or  from  the  operational  parameters  of  the 
authoring  or  delivery  system.  For  example,  inserting  additional  steps  in  a  troubleshooting 
part-task  simulation  looked  like  a  simple  change  in  screens.  However,  it  was  a  very 
complex  and  cumbersome  task  because  of  the  programming  design  required  by  the 
parameters  of  the  authoring  system.  Stated  succinctly;  simple  isn’t  easy,  and  on-site 
contractor  representatives  may  be  necessary  to  explain  the  options  available. 

It  was  a  great  inconvenience  for  Army  reviewers  to  have  to  review  the  first  lessons 
without  the  video  integrated.  Since  early  lesson  production  was  concurrent  with  video 
production,  this  situation  was  virtually  unavoidable,  and  the  timing  of  video  integration 
often  poses  problems  with  CBT  production  projects.  An  alternative  would  be  to  deliver  for 
review  first  those  lessons  which  rely  mainly  on  computer  generated  graphics.  But  this 
program  was  intended  to  rely  heavily  on  interactive  video  to  present  the  new  equipment  to 
the  soldiers  and  most  of  the  templates  used  extensive  video.  The  two  templates  that  used 
mainly  computer-generated  graphics  instead  of  video  (Troubleshooting  and 
Troubleshooting  Introductions)  were  not  designed  until  later  in  the  project.  Another 
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alternative  is  to  approve  the  generic  template  design  or  instructional  approaches  with  a 
prototype  lesson  and  postpone  review  of  lessons  until  video  is  available. 

Another  lesson  learned  during  the  course  of  this  program  was  to  program  a  review 
mode  into  the  lessons.  Contractor  personnel  initially  encountering  frustration  and 
difficulty  in  locating  the  pages  to  which  reviewer  referred  in  comments  on  a  lesson  with 
four  hundred  screens.  The  senior  programmer  then  added  to  the  troubleshooting  template 
a  review  mode  which  displayed  an  identifying  screen  number  for  the  reviewer  to  record 
with  comments  on  the  screen  or  interaction.  This  made  it  easier  for  contractor  personnel 
to  locate  and  correct  the  error,  and  easier  for  the  Government  to  locate  and  verify  that  the 
correction  had  been  made  in  the  final  delivery. 

Recommendations  are  Given  for  Use  of  Templates  for  Courseware  Production. 

There  are  some  limitations  in  the  general  application  of  the  template  approach  to 
courseware  development.  Complex  templates  can  be  developed  for  a  program  cost- 
effectively  only  if  the  program  is  fairly  large.  The  design  and  coding  done  for  the  templates 
could  be  spread  over  many  lessons  on  this  program.  If  existing  templates  are  not  available 
for  smaller  projects,  it  may  be  better  to  design  and  develop  the  lessons  individually,  or  to 
develop  reusable  routines  on  a  smaller  scale  than  templates.  Even  on  large  projects,  it  is 
important  to  analyze  the  training  objectives  to  see  if  they  do  fall  into  categories  before 
attempting  to  develop  templates.  If  there  are  several  types  of  lessons,  several  templates 
may  be  required. 

The  effectiveness  of  lessons  produced  with  templates  depends  on  the 
appropriateness  of  the  template’s  instructional  strategy  and  the  quality  of  its  coding.  The 
use  of  templates  does  not,  by  itself,  guarantee  high  quality  courseware.  If  the  strategies  are 
not  well  designed  or  the  program  is  not  well  coded,  the  courseware  will  not  be  effective  in 
training  the  objectives.  If  the  strategies  of  a  template  are  not  appropriate  for  a  particular 
objective,  the  template  will  not  provide  an  effective  framework  for  the  content. 

Templates  do  not  necessarily  have  to  be  complex  strategies,  but  the  MTP-RC 
statement  of  work  called  for  designs  which  utilized  and  tested  the  capabilities  of  computer- 
based  training  to  provide  complex  and  interesting  training  interactions.  The  use  of 
instructional  templates  which  combine  instructional  strategies  and  program  code  resulted 
in  consistent  user  interface  and  uniform  quality  throughout  the  course. 

Several  lessons  were  learned  about  the  production  process.  The  templates  for  this 
research  program  required  extensive  time  to  develop  because  they  employed  sophisticated 
design  techniques  and  complex  computer  interactions.  New  complex  templates  can  be 
developed  cost  effectively  only  for  fairly  large  programs  which  will  have  several  lessons  of  a 
type.  It  is  suggested  that  ten  lessons  is  a  minimum  number  to  justify  complex  templates. 
Even  on  large  projects,  it  is  important  to  analyze  the  training  objectives  carefully  to  see  if 
they  do  fall  into  categories  before  attempting  to  develop  templates  for  them.  Templates  do 
not  have  to  be  complex,  but  the  MTP-RC  statement  of  work  called  for  designs  which 
utilized  and  tested  the  capabilities  of  computer-based  training  to  provide  complex  and 
interesting  training  interactions.  An  alternative  for  programs  that  are  smaller  or  have 
fewer  occurrences  of  generic  objectives  is  to  develop  reusable  routines  that  handle 
recurring  types  of  interactions,  such  as  answer  judging,  video  loops,  or  standard  feedback. 


It  was  felt  that  the  use  of  instructional  templates  contributed  to  the  overall  quality  of 
the  courseware,  because  they  laid  down  effective  strategies  that  were  uniformly  applied  "to 
many  lessons.  The  use  of  templates  does  help  to  assure  uniform  quality  of  courseware,  but 
templates  alone  do  not  assure  high  quality  courseware.  Individual  lessons  are  likely  to  be 
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good  only  if  the  generic  strategy  is  good;  a  poor  design  repeated  several  times  is  still  a  poor 
design.  Design  of  templates  requires  more  attention  than  design  of  a  single  lesson. 

Another  difficulty  experienced  in  production  was  in  subcontracting  part  of  the 
production  effort  on  this  experimental  program.  The  approach  of  comprehensive 
instructional  templates  required  extensive  communication  between  instructional  designers, 
courseware  programmers,  and  subject  matter  experts.  When  some  of  the  lessons  were 
under  production  at  a  subcontractor’s  site,  communication  about  templates  was  difficult, 
and  the  templates  were  not  implemented  uniformly  at  first.  The  lack  of  SMEs  at  the 
subcontractor  site  was  an  especially  limiting  factor  in  assigning  production  efforts. 


FIELDING  AND  IMPLEMENTATION 

When  the  courseware  was  completed,  three  MicroTICCIT  systems  were  installed  in 
RC  units.  Soldiers  in  the  appropriate  MOSs  were  enrolled  as  subjects  in  the  evaluation 
and  students  on  the  computers.  System  operators  were  trained  at  each  site  to  run  the 
computer  to  make  the  training  available  to  the  soldiers.  To  evaluate  the  effectiveness  of 
the  training  a  pretest  and  a  posttest  were  administered  to  each  soldier  in  the  study.  This 
section  of  the  report  presents  the  experience  gained  in  delivering  computer-based  training 
at  the  RC  units  and  resulting  recommendation  for  fielding. 

Systems  were  Fielded  at  Three  Reserve  Comnonent  Units. 


The  sites  for  the  fielding  were  selected  by  ARI  and  TRADOC  in  consideration  of 
several  factors.  For  the  evaluation  of  the  effectiveness  of  the  training,  a  representative 
sample  of  soldiers  in  the  four  MOSs  was  required.  Units  with  these  MOSs  were  selected  if 
they  would  have  responsibilities  for  Ml  maintenance  upon  mobilization.  Units  connected 
with  divisions  with  which  Mis  had  been  or  would  soon  be  fielded  were  sought.  For  these 
units,  computer-based  training  could  meet  a  real  and  present  need  for  Ml  training.  A 
survey  of  RC  maintenance  units  was  conducted  by  the  contractor  to  provide  information 
for  this  decision.  Two  additional  factors  were  used  in  selecting  sites.  Units  located  in  the 
eastern  part  of  the  country  and  reasonably  close  to  sites  with  Mis  (such  as  the  Ordnance 
School  and  Fort  Bragg)  were  selected  to  reduce  expenses  for  travel  of  contractor 
personnel.  The  finalfactor  was  willingness  of  the  unit  officers  to  cooperate  with  the 
administrative  activities  involved  with  this  trial  fielding. 

The  units  selected  and  the  dates  when  the  computer-based  training  was  available 
are  presented  in  Table  2.  The  training  was  available  from  the  beginning  of  the  fielding  to 
the  evaluation  posttest. 

The  units  were  informed  of  their  selection  during  the  first  year  of  the  program,  and 
each  site  was  visited  by  the  contractor  and/or  ARI  personnel  at  least  once  while  the 
courseware  was  being  produced.  In  early  1986,  formal  briefings  were  held  at  each  unit  for 
the  State  Training  Officer,  unit  Commanding  Officer,  and  unit  Training  Officer  to  acquaint 
them  with  the  purposes  and  procedures  of  the  program  and  to  coordinate  personnel  and 
equipment  needs  for  the  trial  fielding.  The  systems  used  to  deliver  the  courseware 
consisted  of  a  host  minicomputer  and  three  to  six  soldiers  workstations,  which  included  a 
microcomputer,  a  video  screen,  and  a  videodisc  player.  All  units  made  available 
appropriate  facilities  to  the  program,  either  assigning  current  classrooms  or  remodeling 
anotner  room. 
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Table  2 

Fielding  sites  and  training  dates 


195th  Heavy  Equipment  Maintenance  Company 
(General  Support) 

Westminster,  Maryland 

2198th  Heavy  Equipment  Maintenance  Company 
(General  Support) 

Dagsboro,  Delaware 

Detachment  1,  2nd  Battalion,  252nd  Armor 
(Organizational  Support) 

Rea  Springs,  North  Carolina 


Dates  used  for  training 
March,  86  -  June,  87 


April,  86-  June,  87 


August,  86  -  August,  87 


The  195th  is  a  U.S.  Army  Reserve  unit,  and  the  2198th  and  the  2-252nd  are  National 
Guard  units.  Most  RC  maintenance  units  are  Direct  Support  (DS)  or  General  Support 
(GS)  units.  The  2-252nd  is  an  organizational  unit,  assigned  to  round-out  a  division  from 
Fort  Hood. 

Each  site  received  a  complete  MTP-RC  system,  consisting  of  a  host  computer, 
individual  soldier  workstations  connected  by  a  network  to  the  host  computer,  and  the 
program  code  for  the  courseware,  installed  on  the  system.  Each  student  workstation 
consisted  of  a  microcomputer  connected  to  the  host,  a  video  display  screen,  and  a  videodisc 
player  controlled  by  the  microcomputer.  The  number  of  workstations  at  the  sites  varied 
from  three  to  six.  Workstations  were  sometimes  moved  between  sites  to  meet  the  training 
schedules. 

Each  unit  appointed  a  system  operator  to  be  responsible  for  routine  operations  on 
the  computer,  such  as  bringing  the  system  up,  loading  courseware,  registering  soldiers  in 
the  computerized  course  management  program,  performing  routine  computer  maintenance 
procedures,  diagnosing  minor  computer  problems,  and  assisting  soldiers  with  difficulties. 
These  routine  tasks  are  required  to  keep  the  systems  operational.  Each  system  required  an 
operator  to  perform  these  duties,  even  though  the  courseware  was  designed  so  soldiers  as 
students  needed  no  background  or  previous  experience  with  computers  to  use  the  training 
program.  As  the  systems  were  installed  at  the  units,  at  least  two  soldiers  from  each  unit 
were  trained  as  system  operators  by  the  contractor. 

During  the  training  period,  the  contractor’s  field  representative  visited  each  unit 
every  4-6  weeks  to  report  on  system  usage,  assist  site  operators  with  difficulties,  coordinate 
corrections  of  courseware  errors,  and  collect  usage  data.  Both  the  field  representative  and 
the  contractors’  courseware  development  staff  were  on  call  to  assist  the  operators  at  all 
times,  including  during  weekend  drills. 

System  Use  bv  Soldiers  is  Reported. 

Once  the  systems  were  installed,  the  soldiers  began  using  them  to  receive  training 
during  weekend  drills  and  extra,  week  night  drills.  At  the  195th  company,  some  soldiers 
also  spent  a  portion  of  their  annual  training  time  on  the  MTP-RC  system.  Several  soldiers 
who  were  not  a  registered  as  subjects  in  the  study  (either  because  they  joined  the  unit  after 
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the  study  began  or  because  they  did  not  have  one  of  the  4  MOSs)  also  used  the  courseware 
to  learn  about  the  tank.  Usage  of  the  system  is  shown  in  Table  3  for  subjects  in  the  study. 


Table  3 

Total  hours  of  computer  use  by  groups  of  soldiers  in  each  MOS  at  fielding  sites 


A  major  difficulty  encountered  at  all  sites  was  the  limited  time  available  for  soldiers 
to  spend  on  a  research  project.  The  Reserve  Component  soldier  faces  a  heavy  training 
schedule  for  both  individual  task  training  and  unit  tasks.  Unit  commanders  and  training 
officers  find  it  difficult  to  achieve  all  mandated  training  goals  in  the  time  available. 
Although  the  MTP-RC  courses  were  directly  related  to  MOS  responsibilities  of  the 
soldiers,  they  were  not  approved  as  MOS-producing  courses  and  did  not  count  toward  on- 
the-job  training.  With  so  many  demands  on  their  allotted  training  time  already,  soldiers 
understandably  could  not  devote  time  to  this  project  even  when  they  were  interested  in  the 
content.  In  addition,  some  training  activities  require  that  the  units  drill  offsite.  In  those 
cases,  a  soldier  had  to  miss  the  unit  training  and  assemblies  to  take  the  individual  training 
on  the  computer. 

For  the  purposes  of  this  research  project,  the  state-level  National  Guard  commands 
were  able  to  allocate  additional  training  funds  to  pay  soldiers  to  come  in  on  extra  time  to 
take  the  training  at  the  2-252nd  and  the  2198th.  This  made  more  time  available  because 
the  MTP-RC  training  did  not  take  time  away  from  other  requirements.  Financial 
limitations  will  probably  prohibit  this  approach  for  a  full-scale  fielding  of  CBT  training. 
Programs  to  field  this  courseware  or  other  CBT  programs  will  have  to  face  the  problem 
facing  all  training  for  the  Reserve  Component-the  shortage  of  time  for  the  many  training 
requirements.  To  the  extent  that  CBT  can  deliver  effective  training  in  less  time  than  other 
training  media,  it  can  help  save  time. 


The  MTP-RC  design  team  tried  to  design  the  course  lessons  and  simulations  into 
four-hour  modules  to  fit  the  typical  time  period  allotted  for  individual  training  during  an 
RC  weekend  drill,  as  requested  by  the  statement  of  work.  In  fact,  soldiers  seldom  have 
four  hours  straight  for  training  during  a  typical  drill  period.  Most  of  the  knowledge  lessons 
were  short  enough  that  they  could  be  fit  into  shorter  periods.  However,  the  part-task 
simulations  of  lengthy  procedures  usually  required  at  least  as  much  time  as  the  real 
procedure-sometimes  as  long  as  five  or  six  hours.  When  possible,  simulations  should  be 
broken  into  shorter  sections.  For  procedures  where  continued  attention  to  detail  over  a 
long  period  is  critical  to  correct  performance,  then  training  on  the  simulation  should  be 
regarded  as  worthy  of  the  time  required. 

Command  Emphasis  is  Essential  for  Success. 

Command  emphasis  is  essential  for  success  of  a  training  program.  This  program 
was  initiated  to  assist  RC  units  in  meeting  their  training  problems.  CBT  programs  like 
MTP-RC  can  provide  part-task  simulations  to  help  overcome  the  lack  of  new  equipment 
for  training.  CBT  cannot,  however,  overcome  the  lack  of  time  available  for  training  on  all 
tasks.  Even  when  soldiers  are  eager  to  learn  about  new  equipment,  as  many  were  with  this 
Ml  training,  there  often  is  no  time  for  it  after  regular  maintenance  duties,  group  training, 
assemblies,  etc.  Only  if  state  and  unit  commanders  emphasize  and  support  a  training  focus 
will  soldiers  be  able  to  have  time  for  it.  Commanders  must  be  thoroughly  briefed  and  kept 
informed  on  the  status  of  a  program. 

As  a  research  project,  MTP-RC  fielded  training  materials  that  were  not  yet 
validated  as  on-the-job  training  or  MOS-producing  courses.  The  lack  of  official  credit  for 
the  training  sometimes  made  soldiers  and  commanders  reluctant  to  allocate  precious 
training  time  to  the  project.  Computer-based  training  will  be  more  readily  integrated  into 
the  unit’  training  program  if  it  carries  official  status. 


Computer-based  training  on  networked  computer  systems  such  as  the  MTP-RC 
system  has  generally  been  implemented  at  full-time  training  facilities  such  as  service 
schools  with  large  student  populations  and  full-time  training  staff.  In  these  situations,  there 
was  generally  a  staff  person  or  persons  assigned  as  operator  of  the  system,  with 
responsibility  for  performing  routine  maintenance  and  operations,  assisting  students,  and 
supervising  the  computerized  record-keeping  system.  The  Reserve  Component  units  had 
fewer  students  than  schools  ,  but  also  fewer  staff  to  support  training  activities.  There  was 
concern  that  the  RC  personnel  might  not  be  able  to  handle  the  computer. 

The  MTP-RC  experience  was  that  the  CBT  was  not  a  "turnkey"  system.  All 
operators  required  continued  support  after  their  training.  With  the  contractor’s  support, 
the  soldiers  responsible  for  the  systems  were  able  to  handle  most  operator’s  responsibilities 
after  the  three-day  training  program.  At  one  site,  the  operators’  hesitation  in  asking  for 
support  resulted  in  a  delay  of  making  the  second  tape  set  of  the  courseware  available  to 
soldiers  at  the  site. 

An  area  in  which  support  to  the  operators  was  critical  was  maintenance  of  the 
hardware  and  software.  Generally,  the  operators  were  only  stymied  when  they 
encountered  bugs  in  the  courseware  or  unusual  system  malfunctions.  The  field 
representative  assisted  in  periodic  maintenance  and  helped  the  operators  determine  if 
difficulties  were  being  caused  by  hardware  malfunctions  that  could  only  be  recognized  after 
long  experience.  In  these  instances,  contractor’s  support  personnel  were  usually  able  to 
provide  assistance  over  the  telephone. 

Each  of  the  three  systems  required  replacement  of  its  data  storage  disk  once  during 
the  three-year  program.  Diagnosis  and  repair  of  this  problem  required  2-5  days  each  time. 
Training  was  prevented  for  a  full  weekend  at  one  site  because  of  the  disk  failure.  The 
computer  boards  which  created  the  video  displays  for  the  workstations  also  had  difficulties. 
These  "video  overlay"  boards  were  extremely  sensitive  to  physical  shocks  and  to  ambient 
temperatures  over  75  degrees  F,  and  repair  of  these  boards  often  required  3-4  months.  To 
keep  workstations  on-line,  boards  were  sometimes  exchanged  shared  between  units.  Even 
when  individual  workstations  were  down  for  repair  of  the  video  overlay  boards,  the  systems 
as  a  whole  were  available  for  use  on  the  other  workstations.  Some  other  hardware  and 
software  problems  had  disrupted  the  courseware  production  process,  but  did  not  hamper 
delivery. 

System  operators  should  be  trained  and  supported.  Ideally,  the  operator  should  be 
permanently  assigned  to  gain  the  advantage  of  experience  with  the  system.  The  special 
MicroTICCIT  system  coirfigured  for  this  fielding  reduced  the  level  of  skill  and  training 
required  of  the  on-site  operator  for  routine  operations,  but  support  was  required  to  keep 
the  system  up.  The  simplified  courseware  loading  operations  described  earlier  reduced  the 
demands  on  the  operator’s  time  and  technical  competence.  Complex  computer  systems 
offer  sophisticated  training  capabilities,  but  require  more  attention  than  older,  more 
familiar  media.  It  is  recommended  that  a  travelling  field  representative  be  provided  to 
offer  assistance  for  non-routine  problems,  refresher  training,  and  training  for  replacement 
personnel.  This  support  staff  should  have  more  extensive  knowledge  of  the  computer 
system  and  the  courseware  syllabus. 


During  the  development  phase,  it  became  clear  that  the  extensive  graphic  required 
for  the  courseware,  especially  the  troubleshooting  simulations,  would  exceed  the  storage 
capacity  of  the  MicroTICCIT  systems  available  for  the  program.  The  three  systems  had 
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storage  capacities  of  50,  65,  and  100  Megabytes  of  data,  and  the  courseware  was  estimated 
to  require  130-140  Megabytes  of  storage.  It  was  felt  that  if  the  site  operator,  usually  the 
unit’s  training  officer,  would  have  to  continually  load,  process,  and  unload  lessons  to  meet 
the  changing  needs  of  the  unit,  the  time  demand  would  be  excessive  and  the  entire  program 
would  be  jeopardized. 

A  special  configuration  of  the  MicroTICCIT  system  was  developed  by  the 
contractor  with  technical  assistance  from  Hazeltine.  With  this  system,  each  site  received  2 
to  3  sets  of  computer  tapes,  each  of  which  contained  one-third  to  one-half  of  the  lessons 
required  at  that  site.  With  these  special  tape  sets,  the  operator  could  change  the  online 
courseware,  which  might  be  as  much  as  thirty  lessons,  in  one  hour.  Without  such  tape  sets, 
changing  each  lesson  would  require  up  to  five  hours  each.  A  disadvantage  of  the  tape  set 
for  delivering  lessons  is  the  decreased  flexibility  in  making  available  lessons  from  different 
parts  of  different  courses  to  students  depending  on  how  much  of  the  course  they  had 
finished.  Since  it  was  expected  that  for  this  trial  fielding,  most  of  the  soldiers  would  start 
the  program  at  about  the  same  time  and  all  had  about  the  same  length  of  time  available  for 
completion,  the  best  option  was  to  use  the  special  courseware  tape  sets.  Another 
disadvantage  of  the  tape  set  method  used  to  deliver  the  courseware  is  that  it  becomes  more 
difficult  to  make  changes  or  updates  in  the  courseware.  Another  disadvantage  of  the  tape 
set  scheme  was  the  time  required  by  the  contractor  to  modify  the  MicroTICCIT  operation 
system  and  produce  the  tape  sets.  Still,  it  was  considered  better  to  take  a  lot  of  contractor 
operator  time  than  to  require  a  prohibitive  amount  of  on-site  operator  time. 

In  selecting  and  configuring  the  delivery  system  for  the  field,  it  is  important  to 
choose  one  with  a  large  enough  data  storage  capacity  to  handle  the  courseware  needs.  It 
may  be  necessary  to  design  the  courseware  and  get  a  rough  size  estimate  before  deciding 
on  the  system.  Many  systems  are  available  with  varying  storage  capacities.  Larger  capacity 
can  allow  delivery  of  more  programs  on  one  system  and  create  more  flexibility  for 
scheduling  training  for  soldiers.  Systems  such  as  the  Electronic  Information  Delivery 
System  (EIDS)  allow  flexibility  by  simply  changing  diskettes  in  the  computer  and  videodiscs 
in  the  disc  player.  New  storage  technologies  are  emerging  which  use  the  videodisc  as  a 
storage  medium  for  programs  and  data,  as  well  as  video  and  graphic  images.  This 
approach  may  overcome  some  of  the  storage  limitations  currently  faced  in  delivering 
complex  part-task  simulations. 

A  Suitable  Hardware  Environment  must  be  Prepared. 

Some  modifications  or  improvements  to  the  facilities  at  some  units  are 
recommended  to  keep  the  computers  functioning  smoothly  and  reliably.  These 
improvements  are  air  conditioning  and  electrical  system  protectors.  Only  the  195th  lacked 
air  conditioning  in  the  computer  room  by  the  time  of  fielding.  In  spite  of  ambient 
temperatures  over  85  degrees,  the  computer  did  not  malfunction.  Soldiers,  however, 
experienced  discomfort  in  the  environment.  For  installation  of  CBT  systems  which  use 
large  minicomputers,  air  conditioning  is  recommended. 

Electrical  circuits  to  the  computer  room  should  be  adequate.  Ideally,  the  computer 
room  will  have  a  dedicated  branch  circuit  to  avoid  line  fluctuations  caused  by  other 
equipment  in  the  building.  A  Surge  protector  should  be  installed  to  protect  the  system 
from  large  voltage  fluctuations,  whicn  can  damage  the  hardware  and  result  in  loss  of 
student  records  and  downtime  for  restarting  or  repairing  the  system.  At  2  units,  the  host 
computer  was  completely  shut  down  after  each  use  to  avoid  damage  from  voltage  surges 
caused  by  lightning  storms  in  the  area.  On  restarting,  the  systems  exhibited  unreliable 
behavior  until  they  had  warmed  up  for  a  day  or  longer.  If  they  had  been  protected  by  surge 
protectors,  they  could  hrve  been  left  running  and  available  at  all  times. 
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The  computer  room  should  also  have  a  telephone  line  for  support  for  the  operator 
for  unusual  problems.  The  support  personnel  often  need  to  discuss  operations  with  the 
operator  during  the  operation.  In  addition,  with  a  modem  hook-up  over  the  telephone,  the 
support  personnel  can  sometimes  diagnose  and  solve  the  problem  for  the  operator. 


EVALUATION  OF  MTP-RC  COURSEWARE 


Two  types  of  training  evaluation  were  conducted  for  this  program.  First,  a 
formative  evaluation  was  conducted  by  the  contractor  to  assess  tne  courseware  design 
before  completing  and  implementing  the  courseware.  The  formative  evaluation  covered 
overall  design  of  the  courseware,  student  interface,  presentation  strategies,  level  of 
technical  content,  and  testing  strategies.  Some  revisions  of  the  design  and  content  were 
made  as  a  result  of  this  evaluation.  After  the  training  system  was  implemented  at  Reserve 
Component  units  and  soldiers  had  completed  all  or  most  of  the  course  for  their  MOSs,  a 
training  and  transfer  evaluation  was  conducted.  This  evaluation  examined:  1)  if  soldiers 
achieved  the  training  objectives  using  the  courseware,  and  2)  if  skills  learned  on  the 
computer  transferred  to  the  tank,  resulting  in  improvements  in  task  performance.  Testing 
for  the  training  and  transfer  evaluation  was  design  and  conducted  by  the  contractor,  and 
the  data  was  analyzed  by  ARI. 

Some  Design  Modifications  were  made  with  Formative  Evaluation  Input. 

Formative  analysis  of  the  courseware  was  conducted  to  determine  the  need  for 
changes  in  instructional  design,  presentation  strategy,  or  content.  Two  types  of  formative 
evaluation  were  conducted  simultaneously  by  the  contractor: 

one-on-one  sessions  in  which  proctors  watched  soldiers  take  representative  lessons 

and  then  discussed  the  lesson  content  and  format  with  the  soldier,  and 

administration  of  a  hands-on  test  before  and  after  the  lesson. 


To  conduct  the  one-on-one  evaluation,  the  contractor  administered  representative 
lessons  to  recent  graduates  of  the  Advanced  Individual  Training  (AIT)  MOS  or  Additional 
Skill  Indentifier  (ASI)  courses  at  the  Ordnance  School  and  Armor  School  in  early  1985. 
Students  were  observed  as  they  went  through  the  courseware,  and  notes  were  taken 
whenever  students  seemed  to  have  difficulties  with  the  presentation,  design,  or  content  of 
the  lessons.  Sometimes  soldiers  were  interrupted  and  asked  what  their  difficulty  was  or 
why  they  were  hesitating.  A  questionnaire  about  the  course  was  administered,  and  each 
student  was  debriefed  to  determine  attitudes  toward  the  courseware. 

Graham,  Shlecter,  &  Goldberg  (1986)  evaluated  the  transfer  effect!  eness  of  a 
lesson  from  the  MTP-RC  training  with  soldiers  at  the  Armor  School  Soldiers  who 
received  the  simulated  troubleshooting  training  made  fewer  errors  j.  -.r  time  on  hands-on 
troubleshooting  procedures  than  did  soldiers  trained  under  conventional  methods.  The 
comparisions  were,  however,  limited  by  ceiling  effects.  The  skills  and  knowledge 
developed  in  the  exercises  also  generalized  to  a  troubleshooting  task  not  specifically 
trained  in  the  courseware.  This  generalization  was  attributed  to  the  fact  that  soldiers  were 
trained  and  give  practice  on  properly  using  the  TM  and  test  equipment. 


5* 


i 


After  analysis  of  the  information  gained  from  the  test  scores  and  from  student 
comments,  the  following  conclusions  were  reached  on  design  and  development  issues  that 
were  of  concern. 

o  The  "icons"  or  symbols  used  for  selecting  tools  and  performing  actions 
in  the  simulations  were  too  small  and  too  close  together  for  easy 
selection  with  the  light  pen.  In  general,  the  light  pen  required  an 
unreasonable  precision  in  use.  The  icons  were  made  larger  and  separated. 

o  Soldiers  had  little  previous  practice  with  some  of  the  Ml  test  equipment. 

The  Course  One  material  on  the  test  equipment  was  expanded  with  more 
detailed  explanations  and  practice  on  the  equipment. 

o  The  purposes  and  uses  of  the  icons  were  readily  understood;  most  soldiers 
were  comfortable  with  the  simulations  within  an  hour. 

o  The  design  of  the  troubleshooting  simulations  reinforced  the  habit  of 
carefully  selecting  test  points  on  electronic  equipment.  Therefore, 
the  screen  design  which  required  careful  selection  was  retained. 

o  Soldiers  expressed  that  performance  of  the  troubleshooting  tasks  was  more 
satisfying  after  gaining  the  lessons’s  understanding  of  the  reasons  for  the 
procedure.  Previously,  they  had  simply  followed  the  procedure  without 
knowing  the  reasons. 

o  Soldiers  responded  favorably  to  the  cartoons  used  for  motivation. 

o  Soldiers  had  limited  familiarity  with  the  format  of  the  technical  manuals. 

Nearly  all  soldiers  commented  in  debriefing  that  being  forced  to  follow  the 
manuals  exactly  in  the  simulations  was  good  practice. 


In  general,  the  design  of  the  courseware  and  the  level  of  detail  in  the  presentation  of 
knowledge  about  the  tank  subsystems  was  evaluated  to  be  appropriate  and  likely  to  be 
instructionally  effective  when  implemented.  Changes  as  described  in  the  points  above  were 
incorporated  into  all  lesson  to  which  they  applied.  The  template  approach  to  lesson  coding 
facilitated  many  of  these  changes.  For  example,  changing  the  icons  required  only  that  the 
courseware  developers  change  the  coding  routine  which  created  the  icon*  on  screen,  and 
insert  the  new  routine  once  into  each  troubleshooting  lesson.  In  the  traditional  approach 
to  coding  courseware,  it  would  have  been  necessary  to  reprogram  individually  each  of  the 
hundreds  of  screens  on  which  the  icons  appeared. 

The  Training  Transfer  Evaluation  Showed  Improvements  in  Task  Performance. 

For  the  transfer  evaluation,  the  contractor  and  ARI  developed  and  administered 
written  and  hands-on  tests  of  the  skills  targeted  in  the  training  to  determine  if  soldiers  had 
achieved  the  training  objectives  and  if  the  training  transferred  to  the  tank.  The  written  test 
covered  knowledge  about  the  tank  subsystems,  safety,  and  procedures.  The  hands-on  tests 
covered  troubleshooting  procedural  tasks  trained  in  the  courseware.  Both  tests  were 
administered  as  a  pretest,  including  hands-on  tests  on  Ml  tanks  at  the  Ordnance  School  for 
DS/GS  and  Fort  Bragg  for  organizational  in  April,  1986.  The  posttests  were  administered 
in  July  and  August,  1987,  at  the  same  sites.  Soldiers  took  the  tests  at  the  test  site  closest  to 
their  units. 
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The  procedures  and  results  of  the  transfer  evaluation  were  reported  by  Graham 
(1987).  For  the  hands-on  testing,  soldiers  performed  on  the  tank  truncated  troubleshooting 
procedures  from  the  MTP-RC  course.  Trained  monitors  rated  the  soldiers  GO  or  NO  GO~ 
on  each  major  block  of  theprocedures.  If  all  blocks  were  GO,  the  soldiers  received  a  GO 
for  the  entire  procedure.  Tne  primary  performance  measure  was  percent  GOs  on  the 
hands-on  procedures.  Figure  2  shows  that  all  four  MOSs  showed  marked  improvement 
from  the  pretest  to  the  posttest.  Combined  across  MOSs,  performance  of  the  35  soldiers 
improved  from  a  36%  GO  rate  on  the  pretest  to  an  82%  GO  rate  on  the  posttest.  Since  the 
Ml  troubleshooting  procedures  require  exact  performance  of  the  steps  in  sequence  to 
correctly  diagnose  faults,  this  increase  in  error-free  performance  has  practical  importance. 

Analysis  of  the  written  tests  further  indicated  mat  the  MTP-RC  training  was 
effective.  The  written  test  scores  of  the  DS/GS  soldiers  indicated  that  their  Ml  tank 
knowledge  and  skills  were  considerably  below  levels  of  recent  AIT  graduates  and  that 
MTP-RC  training  helped  reduce  that  difference.  The  written  scores  suggested  that  the  job 
knowledge  and  skills  of  the  organizational  soldiers  are  at  about  the  same  level  as  recent 
AIT  graduates,  both  before  and  after  training.  These  soldiers  from  the  2-252  AR  have  Ml 
tanks  in  their  units  and  routinely  maintain  them.  Detailed  description  of  the  evaluation 
and  analysis  of  the  results  were  presented  by  Graham  (1987). 
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Figure  2:  Hands-on  Pretesta  and  Posttests  for  Four  MOSs 


SUMMARY 


This  report  has  evaluated  the  effectiveness  of  CBT  for  RC  units  and  made 
recommendations  on  producing  and  fielding  CBT. 

The  introduction  of  new,  sophisticated  equipment  to  the  Army  inventory  presents  { 

special  training  challenges  to  Reserve  Component  maintenance  units.  They  must  train  and  j 

sustain  skills  in  troubleshooting  and  maintenance  on  the  equipment,  but  have  little  or  no 
access  to  the  equipment,  limited  expertise  in  the  equipment,  and  limited  time  for  training. 

The  Model  Training  Program  for  Reserve  Component  units  was  a  research  and 
development  program  conceived  to  investigate  the  utility  of  computer-based  training  with 
interactive  videodisc  as  a  medium  for  delivering  training  to  RC  units.  Interactive 
courseware  with  part-task  simulations  was  developed  for  four  MOSs  with  maintenance 
responsibility  for  the  Ml  Abrams  tank  and  fielded  at  three  RC  sites  for  trials. 

Development  of  a  large  program  of  courseware  also  allowed  investigation  into  the  process 
of  development. 

The  evaluation  of  the  MTP-RC  courseware  showed  that  soldier  performance  on 
trained  troubleshooting  and  maintenance  tasks  improved  significantly  after  training  on  the 
courseware.  Although  other  factors  may  have  affected  performance,  it  is  reasonable  to 
conclude  that  the  training  effect  was  real  and  significant.  In  addition,  response  of  the 
soldiers  to  the  courseware  was  positive  and  enthusiastic. 

For  a  CBT  program  to  be  successfully  implemented  at  RC  sites,  several  conditions 
must  be  met.  The  heavy  training  schedule  already  facing  these  units  demands  that  any  new 
training  programs--in  any  medium-must  have  Command  emphasis.  The  MTP-RC 
courseware  was  designed  so  that  soldiers  could  use  the  system  without  prior  experience 
with  computers,  but  a  trained  computer  operator  was  still  required  to  support  the  soldiers. 

The  operator  also  needed  training  and  support  to  perform  required  operations  on  the 
computer. 

In  addition  to  developing  and  evaluating  the  courseware,  this  program  also  included 
an  investigation  of  the  process  of  CBT  development.  A  review  of  literature  on  CBT 
development  and  a  survey  of  over  200  developers  of  CBT  courseware  were  conducted  to 
investigate  methods  of  measuring  courseware  quantity  and  quality.  Many  of  the  statements 
frequently  made  by  survey  respondents  echoed  the  experiences  of  the  MTP-RC  program. 

In  summary,  there  is  no  widely  endorsed  method  of  measuring  interactive 
courseware.  Respondents  stated  that  statements  of  work  for  interactive  courseware 
development  require  more  detailed  descriptions  of  soldier  performance  objectives  and 
desireo  training  features  than  do  statements  for  development  of  traditional  training. 

Various  measures,  such  as  instructional  hours,  objectives,  and  number  of  interactions,  are 
in  use,  but  all  have  limitations.  To  overcome  some  of  the  difficulties  of  specifying 
courseware  procurements,  it  was  recommended  that  the  possibility  of  phased  procurements 
be  studied.  In  a  phased  procurement,  the  training  analysis  and  some  design  would  be  done 
as  a  first,  separate  phase  of  the  work.  With  the  results  of  the  training  analysis,  it  w'ould  be 
possible  to  more  clearly  specify  the  desired  soldier  outcomes,  presentation  features,  and 
amount  of  courseware  needed.  With  this  information,  both  the  Government  and 
contractors  could  be  more  confident  in  the  estimates  of  time  and  cost  for  development,  and 
could  avoid  some  of  the  renegotiation  of  scope  that  developers  reported  as  common. 


Enhancements  to  the  Systems  Approach  to  Training  can  be  helpful  in  defining  the 
design  standards  for  a  CBT  development  project.  Issues  such  as  visual  fidelity  in 
simulations,  student  passing  criteria,  types  of  visual  images  need  to  be  decided  differently 
for  interactive  CBT  than  for  shop  training,  videotape,  or  other  linear  presentation  media. 
For  large  CBT  developmeni  projects,  it  can  be  helpful  to  make  design  decisions  on  a 
course-wide  basis  and  apply  them  for  individual  lessons  with  the  use  of  templates,  which 
are  computer  programs  written  to  implement  instructional  strategies.  If  generic  objectives 
can  be  developed,  templates  can  assure  uniform  application  of  design  decisions  across  all 
lessons  of  the  generic  type.  Although  there  are  some  difficulties  with  designing  and 
reviewing  templates,  the  ability  to  implement  sophisticated  strategies  in  many  lessons  with 
templates  can  increase  the  effectiveness  of  the  courseware. 
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